Cleans up messy data frames by performing several operations. These include among others: cleaning of column names, detecting and removing duplicates, empty records and columns, constant columns, replacing missing values by NA, converting character columns into dates when they contain a certain number of date values, detecting subject IDs with wrong formats, etc.
Arguments
- data
The input data frame or linelist
- params
A list of parameters that define what cleaning operations will be applied on the input data. The default parameters are defined in
cleanepi::default_cleanepi_settings()
. The possible values are:standardize_column_names
: A list with the arguments needed to standardize the column names. The elements of this list will be the input for thestandardize_column_names()
function.replace_missing_values
: A list of parameters to be used when replacing the missing values byNA
. The elements of the list are the inputs for thereplace_missing_values()
function.remove_duplicates
: A list with the arguments that define the columns and other parameters to be considered when looking for duplicates. They are the input values for theremove_duplicates()
function.remove_constants
: A list with the parameters that define whether to remove constant data or not. The values are the input for theremove_constants()
function.standardize_dates
: A list of parameters that will be used to standardize the date values from the input data. They represent the input values for thestandardize_dates()
function.standardize_subject_ids
: A list of parameters that are needed to check the IDs that comply with the expect format. These arguments are the input values of thecheck_subject_ids()
.to_numeric
: a vector of column names. When provided, the values in these columns will be converted into numeric.dictionary
: A data frame that will be used to substitute the current values in the specified columns the those in the dictionary. It is the main argument for theclean_using_dictionary()
function.check_date_sequence
: A list of arguments to be used when determining whether the sequence of date events is respected across all rows of the input data. The value in this list are the input for thecheck_date_sequence()
function.span
: A list with the parameters that define how the time span will be calculated between 2 columns of between a column and a Date value. These arguments will be used in thespan()
function.
Value
The cleaned input date according to the user-specified parameters.
This is associated with a data cleaning report that can be accessed using
attr(cleaned_data, "report")
Examples
# Parameters for column names standardization
standardize_col_names <- list(keep = NULL, rename = NULL)
# parameters to remove constant columns, empty rows and columns
remove_cte <- list(cutoff = 1)
# Parameters for substituting missing values with NA:
replace_missing_values <- list(target_columns = NULL, na_strings = "-99")
# Parameters for duplicates removal across all columns
remove_duplicates <- list(target_columns = NULL)
# Parameters for dates standardization
standardize_dates <- list(target_columns = NULL,
error_tolerance = 0.4,
format = NULL,
timeframe = as.Date(c("1973-05-29",
"2023-05-29")),
orders = list(
world_named_months = c("Ybd", "dby"),
world_digit_months = c("dmy", "Ymd"),
US_formats = c("Omdy", "YOmd")
),
modern_excel = TRUE)
# Parameters for subject IDs standardization
standardize_subject_ids <- list(target_columns = "study_id",
prefix = "PS",
suffix = "P2",
range = c(1, 100),
nchar = 7)
to_numeric <- list(target_columns = "sex", lang = "en")
# dictionary = NULL the dictionary-based cleaning will not be performed here
cleaned_data <- clean_data(
data = readRDS(system.file("extdata", "test_df.RDS",
package = "cleanepi")),
params = list(
standardize_column_names = standardize_col_names,
remove_constants = remove_cte,
replace_missing_values = replace_missing_values,
remove_duplicates = remove_duplicates,
standardize_dates = standardize_dates,
standardize_subject_ids = standardize_subject_ids,
to_numeric = to_numeric,
dictionary = NULL
)
)
#>
#> cleaning column names
#> replacing missing values with NA
#> removing the constant columns, empty rows and columns
#> removing duplicated rows
#> standardising date columns
#> checking subject IDs format
#> Warning: Detected incorrect subject ids at lines: 3, 5, 7
#> Use the correct_subject_ids() function to adjust them.
#> converting sex, en into numeric