Function to perform various methods of microaggregation.
microaggregation(
obj,
variables = NULL,
aggr = 3,
strata_variables = NULL,
method = "mdav",
weights = NULL,
nc = 8,
clustermethod = "clara",
measure = "mean",
trim = 0,
varsort = 1,
transf = "log",
gamma = 0.2,
refine = FALSE
)either an object of class sdcMicroObj-class or a data.frame
variables to microaggregate. For NULL: If obj is of class
sdcMicroObj, all numerical key variables are chosen per default. For
data.frames, all columns are chosen per default.
aggregation level (default=3)
for data.frames, by-variables for applying microaggregation only
within strata defined by the variables. For sdcMicroObj-class-objects, the
stratification-variable defined in slot @strataVar is used. This slot can be changed any
time using strataVar<-.
pca, rmd, onedims, onedimsopt, single, simple, clustpca, pppca, clustpppca, mdav, vmdav, clustmcdpca, influence, mcdpca
sampling weights. If obj is of class sdcMicroObj the vector of sampling weights is chosen automatically. If determined, a weighted version of the aggregation measure is chosen automatically, e.g. weighted median or weighted mean.
number of cluster, if the chosen method performs cluster analysis
clustermethod, if necessary
aggregation statistic, mean, median, trim, onestep (default=mean)
trimming percentage, if measure=trim
variable for sorting, if method=single
transformation for data x
gain factor of method ‘vmdav’ (a single non-negative number,
default 0.2), ignored by all other methods. The larger gamma, the more
readily a group is extended beyond aggr records; see Details.
logical (default FALSE). If TRUE, the groups found by
the method are improved by a local search (moving and swapping records) before
they are aggregated; see Details.
If ‘obj’ was of class sdcMicroObj-class the corresponding
slots are filled, like manipNumVars, risk and utility. If ‘obj’ was
of class “data.frame”, an object of class “micro” with following entities is returned:
x: original data
mx: the microaggregated dataset
method: method
aggr: aggregation level
measure: proximity measure for aggregation
On https://research.cbs.nl/casc/glossary.htm one can found the “official” definition of microaggregation:
Records are grouped based on a proximity measure of variables of interest, and the same small groups of records are used in calculating aggregates for those variables. The aggregates are released instead of the individual record values.
The recommended method is “rmd” which forms the proximity using multivariate distances based on robust methods. It is an extension of the well-known method “mdav”. However, when computational speed is important, method “mdav” is the preferable choice.
While for the proximity measure very different concepts can be used, the aggregation itself is naturally done with the arithmetic mean. Nevertheless, other measures of location can be used for aggregation, especially when the group size for aggregation has been taken higher than 3. Since the median seems to be unsuitable for microaggregation because of being highly robust, other mesures which are included can be chosen. If a complex sample survey is microaggregated, the corresponding sampling weights should be determined to either aggregate the values by the weighted arithmetic mean or the weighted median.
This function contains also a method with which the data can be clustered with a variety of different clustering algorithms. Clustering observations before applying microaggregation might be useful. Note, that the data are automatically standardised before clustering.
The usage of clustering method ‘Mclust’ requires package mclust02, which must be loaded first. The package is not loaded automatically, since the package is not under GPL but comes with a different licence.
The are also some projection methods for microaggregation included. The robust version ‘pppca’ or ‘clustpppca’ (clustering at first) are fast implementations and provide almost everytime the best results.
Univariate statistics are preserved best with the individual ranking method (we called them ‘onedims’, however, often this method is named ‘individual ranking’), but multivariate statistics are strong affected.
With method ‘simple’ one can apply microaggregation directly on the (unsorted) data. It is useful for the comparison with other methods as a benchmark, i.e. replies the question how much better is a sorting of the data before aggregation.
Method ‘vmdav’ (V-MDAV, Solanas and Martinez-Balleste, 2006) forms groups
of variable size. aggr is then the minimum group size \(k\).
The variables are standardised, and until fewer than \(k\) records are left,
a group is built from the record that is most distant from the centroid of the
data and its \(k - 1\) nearest unassigned records. The group is then extended:
let \(e\) be the unassigned record closest to the group, \(d_{in}\) its
distance to the nearest record of the group and \(d_{out}\) its distance to
the nearest other unassigned record. \(e\) joins the group if
\(d_{in} < \gamma d_{out}\), and this is repeated until the
condition fails or the group has \(2k - 1\) records. Records that are left at
the end (fewer than \(k\)) are added to the group with the closest centroid, so
that a group can exceed \(2k - 1\) records by at most \(k - 1\). Every group
has at least \(k\) records. With gamma = 0 no group is extended and the
group size is fixed as for ‘mdav’; the authors used gamma = 0.2 for
scattered data and gamma = 1.1 for data with natural clusters, where the
information loss can be much lower than with a fixed group size. The values are
aggregated with the arithmetic mean. Ties are broken by the order of the
records, so that the result is deterministic. Distances between records with
missing values are computed from the variables observed in both records;
missing values stay missing. The method needs no distance matrix, but its
running time grows quadratically with the number of records.
Method ‘onedimsopt’ is optimal univariate microaggregation (Hansen and
Mukherjee, 2003). Like ‘onedims’ it treats each variable on its own. Instead
of cutting the sorted values into groups of exactly aggr values, it finds
the partition into groups of at least aggr values (\(k\)) with the smallest
within-group sum of squares. Such a partition consists of groups of \(k\) to
\(2k - 1\) neighbours in the sorted values (Domingo-Ferrer and Mateo-Sanz, 2002)
and corresponds to a shortest path, which is found by dynamic programming in
\(O(kn)\) after sorting. For a single variable no partition into groups of at
least \(k\) values loses less information. For several variables the result is
the best individual ranking; as with ‘onedims’ the groups differ from
variable to variable, so that a record is not hidden in one group of \(k\)
records. The groups are replaced by their means, measure, trim and
weights are not used. Missing values stay missing and the observed values
of a variable are grouped among themselves.
With refine = TRUE the groups of a method are improved by a local search
before they are aggregated. The variables are standardised and the records are
visited in turn. A record is moved to another group, or it changes places with a
record of another group, if this lowers the within-group sum of squares; of all
such changes for the record the best one is made. A group never falls below
aggr records, and no record is moved into a group that already has
\(2k - 1\) records. The search ends when no move and no swap lowers the sum of
squares any more. The result is a local optimum, in general not the global one,
and its information loss is never higher than without the refinement. A local
search that improves a feasible partition step by step is also the building block
of the iterated local search of Laszlo and Mukherjee (2015). The refined groups
are replaced by their unweighted means, whatever measure and
weights are. One pass over the
records costs \(O(n^2 p)\) for \(n\) records and \(p\) variables. The
refinement is not available for ‘onedims’ and ‘onedimsopt’, for a
single variable (except with ‘vmdav’) and for data with missing values.
if only one variable is specified, mafast is applied and argument method is ignored,
except for the methods vmdav and onedimsopt, which are also applied to a single variable.
Parameters measure are ignored for methods mdav, vmdav, onedimsopt and rmd.
Domingo-Ferrer, J. and Mateo-Sanz, J.M., Practical Data-Oriented Microaggregation for Statistical Disclosure Control, IEEE Transactions on Knowledge and Data Engineering, 14 (1), pp. 189-201, 2002.
Hansen, S.L. and Mukherjee, S., A Polynomial Algorithm for Optimal Univariate Microaggregation, IEEE Transactions on Knowledge and Data Engineering, 15 (4), pp. 1043-1044, 2003. doi:10.1109/TKDE.2003.1209020
Laszlo, M. and Mukherjee, S., Iterated Local Search for Microaggregation, Journal of Systems and Software, 100, pp. 15-26, 2015. doi:10.1016/j.jss.2014.10.012
Solanas, A. and Martinez-Balleste, A., V-MDAV: A Multivariate Microaggregation With Variable Group Size, in COMPSTAT 2006, Rome, pp. 917-925, 2006.
Templ, M. and Meindl, B., Robust Statistics Meets SDC: New Disclosure Risk Measures for Continuous Microdata Masking, Lecture Notes in Computer Science, Privacy in Statistical Databases, vol. 5262, pp. 113-126, 2008.
Templ, M. Statistical Disclosure Control for Microdata Using the R-Package sdcMicro, Transactions on Data Privacy, vol. 1, number 2, pp. 67-85, 2008. http://www.tdp.cat/issues/abs.a004a08.php
Templ, M. New Developments in Statistical Disclosure Control and Imputation: Robust Statistics Applied to Official Statistics, Suedwestdeutscher Verlag fuer Hochschulschriften, 2009, ISBN: 3838108280, 264 pages.
Templ, M. Statistical Disclosure Control for Microdata: Methods and Applications in R. Springer International Publishing, 287 pages, 2017. ISBN 978-3-319-50272-4. doi:10.1007/978-3-319-50272-4 doi:10.1007/978-3-319-50272-4
Templ, M. and Meindl, B. and Kowarik, A.: Statistical Disclosure Control for Micro-Data Using the R Package sdcMicro, Journal of Statistical Software, 67 (4), 1–36, 2015.
data(testdata)
# donttest since Examples with CPU time larger 2.5 times elapsed time, because
# of using data.table and multicore computation.
# \donttest{
m <- microaggregation(
obj = testdata[1:100, c("expend", "income", "savings")],
method = "mdav",
aggr = 4
)
summary(m)
#> $meansx
#> expend income savings
#> Min. : 1106874 Min. : 2897 Min. : 11751
#> 1st Qu.:25977689 1st Qu.:27750000 1st Qu.:2620342
#> Median :45716872 Median :44850000 Median :4771488
#> Mean :48440371 Mean :49180278 Mean :4798498
#> 3rd Qu.:69426340 3rd Qu.:70650000 3rd Qu.:6940269
#> Max. :98685205 Max. :99600000 Max. :9984098
#>
#> $meansxm
#> expend income savings
#> Min. :14471827 Min. : 4482460 Min. : 872137
#> 1st Qu.:22752145 1st Qu.:24675000 1st Qu.:2353262
#> Median :42916487 Median :46850000 Median :5116959
#> Mean :48440371 Mean :49180278 Mean :4798498
#> 3rd Qu.:71888065 3rd Qu.:65000000 3rd Qu.:7020042
#> Max. :91918606 Max. :93725000 Max. :9407783
#>
#> $amean
#> [1] 0
#>
#> $amedian
#> [1] 0.1782512
#>
#> $aonestep
#> [1] 0
#>
#> $devvar
#> [1] 0.3747106
#>
#> $amad
#> [1] 0.5343033
#>
#> $acov
#> [1] 0.1873553
#>
#> $arcov
#> [1] NA
#>
#> $acor
#> [1] 0.25935
#>
#> $arcor
#> [1] NA
#>
#> $acors
#> [1] 0.6424174
#>
#> $adlm
#> [1] 0.1699611
#>
#> $adlts
#> [1] NA
#>
#> $apcaload
#> [1] 0.6853318
#>
#> $apppcaload
#> [1] 2.27684
#>
#> $totalsOrig
#> expend income savings
#> 4844037117 4918027777 479849813
#>
#> $totalsMicro
#> numeric(0)
#>
#> $atotals
#> [1] 0
#>
#> $pmtotals
#> [1] 0
#>
#> $util1
#> [1] 53.39995
#>
#> $deigenvalues
#> [1] 0.0596172
#>
#> $risk0
#> [1] 0
#>
#> $risk1
#> [1] 0.32
#>
#> $risk2
#> [1] 0
#>
#> $wrisk1
#> [1] 0.9830069
#>
#> $wrisk2
#> [1] 0
#>
## variable group size (V-MDAV): aggr is the minimum group size
mv <- microaggregation(
obj = testdata[1:100, c("expend", "income", "savings")],
method = "vmdav",
aggr = 4,
gamma = 1.1
)
table(table(mv$mx$expend))
#>
#> 4 5 6 7
#> 7 4 4 4
## refinement step: the groups of a method are improved by a local search
mr <- microaggregation(
obj = testdata[1:100, c("expend", "income", "savings")],
method = "mdav",
aggr = 4,
refine = TRUE
)
# sum of squared differences to the original values, standardised variables
s <- apply(m$x, 2, sd)
c(mdav = sum(scale(m$x - m$mx, FALSE, s)^2),
refined = sum(scale(mr$x - mr$mx, FALSE, s)^2))
#> mdav refined
#> 27.88599 25.53098
## optimal univariate microaggregation of each variable
mo <- microaggregation(
obj = testdata[1:100, c("expend", "income", "savings")],
method = "onedimsopt",
aggr = 4
)
table(table(mo$mx$expend))
#>
#> 4 5 6 7
#> 18 3 1 1
## for objects of class sdcMicro:
## no stratification because `@strataVar` is `NULL`
data(testdata2)
sdc <- createSdcObj(
dat = testdata2,
keyVars = c("urbrur", "roof", "walls", "water", "electcon", "sex"),
numVars = c("expend", "income", "savings"),
w = "sampling_weight"
)
sdc <- microaggregation(
obj = sdc,
variables = c("expend", "income")
)
## with stratification using variable `"relat"`
strataVar(sdc) <- "relat"
sdc <- microaggregation(
obj = sdc,
variables = "savings"
)
# }