Resolving the SAS EG Transcoding Error

-
Resolving the SAS EG Transcoding Error

Addressing the "Character Data Lost During Transcoding" Issue in SAS EG

Author: Sarath

Date: November 19, 2024

Introduction

While working in SAS Enterprise Guide (SAS EG), you may encounter the error: "Some character data was lost during transcoding in the dataset." This issue typically arises when character data contains unsupported characters or is truncated due to insufficient column lengths. In this blog post, we'll explore the root causes and provide step-by-step solutions.

Common Causes

  • Unsupported Characters: The data contains special or non-ASCII characters not representable in the session encoding.
  • Truncation: Character variables are too short to store the full data, leading to loss of information.
  • Encoding Mismatch: The dataset's encoding differs from the SAS session's encoding.

Step-by-Step Solutions

1. Check Encoding

Identify the encoding of your SAS session and dataset:

proc options option=encoding; run;
proc contents data=tempdata.cm; run;

2. Identify Problematic Characters

Review a sample of the dataset to locate non-representable or truncated characters:

proc print data=tempdata.cm (obs=50); run;

3. Use a Compatible Encoding

Adjust the encoding for your session or dataset. For example, specify UTF-8 if working with multilingual data:

libname tempdata 'path-to-data' inencoding='utf-8';

4. Increase Column Lengths

Expand the length of character variables to avoid truncation:

data tempdata.cm;
    set tempdata.cm;
    length newvar $200; /* Adjust length */
    newvar = oldvar;
run;

5. Transcode the Dataset

Convert the dataset into a compatible encoding:

libname tempdata_in 'path-to-input-data' inencoding='utf-8';
libname tempdata_out 'path-to-output-data' encoding='utf-8';

data tempdata_out.cm;
    set tempdata_in.cm;
run;

6. Modify Encoding with PROC DATASETS

Repair the dataset’s encoding directly:

proc datasets lib=tempdata;
    modify cm / correctencoding='utf-8';
quit;

7. Clean the Data

Handle non-printable or invalid characters using functions like KCOMPRESS or KSTRIP.

Best Practices

  • Ensure consistent encoding between your data sources and SAS session.
  • Use UTF-8 encoding for handling multilingual data.
  • Allocate sufficient column lengths for character variables during data transformation.

Conclusion

Resolving transcoding errors in SAS EG requires identifying the root cause and applying the appropriate solution. By following the steps outlined above, you can ensure your character data is correctly processed without loss or truncation.

Have questions or insights? Share them in the comments below!

Popular posts from this blog

SAS Interview Questions and Answers: CDISC, SDTM and ADAM etc

-
1) What do you know about CDISC and its standards? CDISC stands for Clinical Data Interchange Standards Consortium and it is developed keeping in mind to bring great deal of efficiency in the entire drug development process. CDISC brings efficiency to the entire drug development process by improving the data quality and speed-up the whole drug development process and to do that CDISC developed a series of standards, which include Operation data Model (ODM), Study data Tabulation Model (SDTM) and the Analysis Data Model ADaM ). 2) Why people these days are more talking about CDSIC and what advantages it brings to the Pharmaceutical Industry? A) Generally speaking, Only about 30% of programming time is used to generate statistical results with SAS®, and the rest of programming time is used to familiarize data structure, check data accuracy, and tabulate/list raw data and statistical results into certain formats. This non-statistical programming time will be significantly reduced...
Read more

Comparing Two Methods for Removing Formats and Informats in SAS: DATA Step vs. PROC DATASETS

-
Comparing Two Approaches to Removing Formats and Informats in SAS Comparing Two Approaches to Removing Formats and Informats in SAS When working with SAS datasets, there are times when you need to remove formats and informats that have been previously assigned to variables. Two primary approaches can be used for this task: Using the DATA Step Using the PROC DATASETS Procedure This article compares and contrasts these two approaches to help you determine which method is most appropriate for your needs. Approach 1: Using the DATA Step The DATA step is a versatile and commonly used method for removing formats and informats. By assigning variables to a null format or informat, you can effectively remove these attributes from your dataset. Example: data mydata_clean; set mydata; format _all_; informat _all_; run; In this example, the mydata dataset is processed in the DATA step, and...
Read more

Studyday calculation ( --DY Variable in SDTM)

-
USE OF THE “STUDY DAY” VARIABLES The permissible Study Day variables (--DY, --STDY, and --ENDY) describe the relative day of the observation starting with the reference date as Day 1. They are determined by comparing the date portion of the respective date/time variables (--DTC, --STDTC, and --ENDTC) to the date portion of the Subject Reference Start Date (RFSTDTC from the Demographics domain). The Subject Reference Start Date (RFSTDTC) is designated as Study Day 1. The Study Day value is incremented by 1 for each date following RFSTDTC. Dates prior to RFSTDTC are decremented by 1, with the date preceding RFSTDTC designated as Study Day -1 (there is no Study Day 0). This algorithm for determining Study Day is consistent with how people typically describe sequential days relative to a fixed reference point, but creates problems if used for mathematical calculations because it does not allow for a Day 0. As such, Study Day is not suited for use in subsequent numerical computations...
Read more