ArrayStructures

Intro

In many cases you would like to either (i) store data which have more than two dimensions or (ii) collect data of different type/dimension under the same variable name. Examples of the former could be interest rates for different maturities for different countries over time or Monte-Carlo simulations for different sample sizes and different model specifications. Examples of the latter could be real time prices grouped on a daily basis (number of observations varies from day to day) or results of OLS estimation stored in one variable. In MATLAB you can address these issues using:

1. Multidimensional arrays
2. Structures/Structure Arrays
3. Cells/Cell arrays

Each of these approaches has its advantages and disadvantages. For example: the easiest way to handle data in a uniform way is to use multidimensional arrays. Multidimensional arrays, however, require that the data are of the same type (logical, numerical, character). In the next sections we will briefly review these data objects and discuss the most straightforward way to handle them.

Multidimensional Arrays

Elementwise Operations

Multidimensional arrays are just a generalization of a matrix. Almost all MATLAB functions can generate and operate with multivariate arrays. For example, to generate a three-dimensional array of standardized normally distributed observations with dimensions [math]T,p,k[/math], you have to type:

  A=randn(T,p,k);

MATLAB prints all multidimensional arrays as a sequence of matrices. Say, [math]2\times 2\times 2[/math] array is printed as

>> A=zeros(2,2,2)
A(:,:,1) =
     0     0
     0     0
A(:,:,2) =
     0     0
     0     0

Taking an average over the second dimension is quite intuitive:

  meanA=mean(A,2);

Taking a maximum over the third dimension is less so:

  maxA=max(A,[],3);

The bottom line is: please check MATLAB help system first.

Multidimensional arrays support element-by-element binary operations +,-,.*,./,.^,<,>,~= between two arrays with the same dimensions and between arrays and scalars. MATLAB will correctly compute

  B=A.^2;
  C=A.^B+B;
  D=C./A;

In the first line, 3-D array B consists of squared values of 3-D array A. In the second line elements of 3-D array C consist of elements of A in power of elements B plus elements from B. 3-D array D consists of element-wise division of C by A. However, the following commands

A*A;
A^2;

will throw an error, since ^,* are matrix operations and they are not defined on multidimensional arrays.

Collapsing Singleton Dimensions

All operations of extracting sub-arrays from the original arrays work as usual (see link). However, there are some particularities that I would like to mention. Assume that we have a [math]3\times 3\times 3[/math] array A. MATLAB recognizes the following operation :

  A(:,:,1)*A(:,:,2);

while

  A(1,:,:)*A(1,:,:);

generates an error. We get similar results for plot function:

  plot(A(:,:,1))

generates a graph, while

  plot(A(1,:,:))

generates an error. It happens because, from MATLAB point of view, A(1,:,:) is not a matrix, but a 3-D array. There are two ways to deal with this problem:

1. Make MATLAB realize that we would like to see a matrix instead of a 3-D array with a singleton in the first dimension

           B(:,:)=A(1,:,:);
           plot(B)

2. Collapse all singleton dimensions using a special command squeeze

           B=squeeze(A(1,:,:));
           plot(B)

Expanding to Higher Dimensions

By default, element-wise operations are defined on any multi-dimensional array of the same dimension. An additional set of operations is defined on conformable vectors and matrices. The natural question to ask is “What to do if you want to subtract a vector from a matrix, or a matrix from N-D array?” MATLAB has a special function that can transform a lower-dimensional array to a higher-dimensional one by replicating the original content:

 A=repmat([1,2,3]',2,3)
A =
     1     1     1
     2     2     2
     3     3     3
     1     1     1
     2     2     2
     3     3     3

This command replicates a column-vector [math][1\ 2\ 3]'[/math] six times, i.e. twice along each row and 3 times along each column. For 3-D replication we have to use slightly different syntax:

   A=repmat([1,2,3]',[1 2 3])
A(:,:,1) =
     1     1
     2     2
     3     3
A(:,:,2) =
     1     1
     2     2
     3     3
A(:,:,3) =
     1     1
     2     2
     3     3

This command keeps the length of the vector the same (first index is 1), replicates it twice along the second dimension (second index is 2) and 3 times along the third dimension (third index is 3). Consider the following two real-life examples:

1. Assume that we have a cross-section of returns and we would like to subtract a series of risk-free rate
2. Assume that we need to subtract a mean along the third dimension from a 3-D array.

These examples can be implemented either by using a loop, or by using repmat. In the first case the algorithm with a loop looks like:

1. Initialize a matrix of excess returns Rex
2. Compute a difference of R(:,i)-rf and assign it to a column Rex(:,i) for [math]i=1[/math]
3. Repeat (2) for i=2, 3, …, size(R,2) times

% rf - a series of risk-free returns
% R  - a cross-section of returns, it is assumed that size(R,1)=size(rf,1)
Rex=zeros(size(R));
for i=1:size(R,2)
Rex(:,i)=R(:,i)-rf;
end

The same algorithm using repmat looks:

1. Replicate rf vector size(R,2) times
2. Compute Rex by Rx=R-rf

% rf - a series of risk-free returns
% R  - a cross-section of returns, it is assumed that size(R,1)=size(rf,1)
Rf=repmat(rf,1,size(R,2));
Rex=R-Rf;

For the second example, to implement an algorithm with a loop:

1. Compute a mean of 3-D array Rmean3
2. Initiate a 3-D array of demeaned values
3. Compute a difference between R(:,:,i)-Rmean3 and assign it to Rdemean(:,:,i) for i=1
4. Repeat (3) for i=2, 3, …, size(R,3)
5. check whether the mean is indeed subtracted

  R=rand(3,4,10); % constructing a 3-D array of U(0,1) random variables
  Rmean3=mean(R,3); %constructing a matrix of means
  Rdemean=zeros(size(R)); %initializing a 3-D array of zeros
  for i=1:size(R,3)
    Rdemean(:,:,i)=R(:,:,i)-Rmean3;
  end
  disp(mean(Rdemean,3)) % which is a matrix of zeros implying we worked correctly

The same algorithm with repmat:

1. Compute a mean of 3-D array Rmean3
2. Construct a 3-D array of means using repmat
3. subtract one from another
4. check whether the mean is indeed subtracted

  R=rand(3,4,10); % constructing a 3-D array of U(0,1) random variables
  Rmean3=mean(R,3); %constructing a matrix of means
  Rmean3expand=repmat(Rmean3,[1 1 size(R,3)]); %Please note, for more than 2-dimensional arrays, repmat accepts a vector of replications instead of a variable number of inputs.
  Rdemean=R-Rmean3expand;
  disp(mean(Rdemean,3)) %matrix of zeros, which mean that we did it correctly

Structures and Cell Arrays

Sometimes it is natural to keep data of different type under the same roof, i.e. using the same variable name. Multidimensional arrays are not designed for this purpose. Therefore structures, arrays of structures or cell arrays have to be used in such cases. Structures and cell arrays are very similar and, in fact, interchangeable. For some applications, however, cell arrays are more suitable while, for other, structures are preferable. For obvious reasons, most of binary operations are not defined on these objects.

Structures

Structure variables are variables that have “fields”. The variable name is separated from the field name by a dot. For example, if you want to keep all OLS regression results, i.e. beta coefficients, covariance matrix, t-stats and vector of residuals in one place, you can do it using the following:

%Assuming X and y are already defined, the whole filling-up process of the structure would look as follows:
OLS.beta=X\y;%short and more efficient way to write inv(X'*X)*X'*y
OLS.resid=y-X*OLS.beta;
[T,k]=size(X);
OLS.sigma2=sum(OLS.resid.^2)/(T-k);%computing residual variance
OLS.cov=OLS.sigma2*inv(X'*X);
OLS.tstat=OLS.beta./sqrt(diag(OLS.cov));
OLS.name='Regression one';

An assignment of OLS variable to another one creates a copy of the structure with all fields and values.

>> OLSnew=OLS;
>> OLSnew.name
ans =
Regression one

A field can be a structure by itself. For example,

>> OLSnew.old=OLS;
>> OLSnew.old.name
ans =
Regression one

Moreover, all field names will be carried in and out of a function. In this way you should not worry about the order of inputs and outputs for a function with many inputs/outputs.

There are several useful functions that are quite helpful for dealing with structures:

• isfield(struct,field_name) checks whether a structure struct has a field name field_name. It returns either 1 (true) or 0 (false)

    >> isfield(OLS,'name')
    ans =
    1
    >> isfield(OLS,'pvalues')
    ans =
    0

• fieldnames(struct) generate a cell array (see Section [cell]) with all field names of a structure struct

  >> fnames=fieldnames(OLSnew)'
  fnames =
      'beta'    'resid'    'sigma2'    'cov'    'tstat'    'name'    'old'

• Indirect referencing. You can access the values of fieldnames using the information generated by fieldnames(struct) using the following syntax:

  >> OLSnew.(fnames{6}) %please note curly brackets!
  ans =
  Regression one

  Since fnames{6}=’name’, we indirectly refer to the field name of the structure OLSnew. In this way we can compare two structures field-by-field.

• Useful commands for working with structure field names are:

  • intersect(A,B), in terms of set operations, it corresponds to [math]A\cap B[/math]

  • union(A,B), corresponds to [math]A\cup B[/math]

  • setdiff(A,B), corresponds to [math]A/B[/math] (please note, [math]A/B \ne B/A[/math])

  • setxor(A,B), corresponds to [math](A/B)\cup (B/A)[/math]

    f1={'first','second'}; %Creating a cell array, see next section for details
    f2={'second','third'}; %Creating a cell array, see next section for details
  >> disp(intersect(f1,f2))
    'second'
  >> disp(union(f1,f2))
    'first'    'second'    'third'
  >> disp(setdiff(f1,f2))
    'first'
  >> disp(setdiff(f2,f1))
    'third'
  >> disp(setxor(f1,f2))
    'first'    'third'

Structures can be collected into arrays. Please note, all member structures of an array have to have the same set of fields. Creating a field for one element of the array automatically creates empty fields of the same name for all elements of your array. However, despite the fact that the field names are the same, there are no restriction on the field types. Example:

>> s(2).f1=1;
>> disp(s(1))
   f1:[];
>> s(2).f2='Sally';
>> disp(s(1))
    f1:[]
    f2:[]
>> s(1).f1=3;
>> s(1).f2=[83   116   117   100   101   110   116 32];

By assigning 1 to s(2).f1, you automatically create

1. the first element of the array s
2. an empty field s(1).f1

By assigning ‘Sally’ to s(2).f2, you automatically create an empty field s(1).f2. You can print all values of a structure field in an array using “:” operator:

>> s(:).f1
ans =
     3
ans =
     1
>> s(:).f1
ans =
     83   116   117   100   101   110   116 32
ans =
     Sally

and try to construct an array out of them using concatenation operator [ ]:

[s(:).f1]
ans =
     3     1

Results, though, may vary (try to figure it out by yourself!):

  [s(:).f2]
ans =
Student Sally

An additional hint:

  [s(:).f2]+0
ans =
    83   116   117   100   101   110   116    32    83    97   108   108   121

Some MATLAB functions require structures as inputs (See non-linear optimization for details). Other MATLAB functions return structures when they are asked to do so. For example, data=load(fname) creates a structure data with fieldnames that coincide with variables of the workspace saved in the data file fname. Thus, for comparing variables from different MATLAB data files, we first need to create two structures

  data1=load(fname1);
  data2=load(fname2);

and then compare the fields of interest.

It is easy to write a code that automatically compares variables with the same names from different data files, using intersect and setdiff commands:

data1=load('data1');%data1.mat has to exist
data2=load('data2');%data2.mat has to exist
if isequal(data1,data2) %the only binary operation defined on structures and cell arrays
    disp('These structures are equal')
else
    fname1=fieldnames(data1);%retrieving field names from data1 structure
    fname2=fieldnames(data2);%retrieving field names from data2 structure
    fnamejoint=intersect(fname1,fname2);%constructing a collection of field names that belong to both structures
    for i=1:length(fnamejoint)
        if isequal(data1.(fnamejoint{i}),data2.(fnamejoint{i})) %indirect referencing
            disp([fnamejoint{i} ' fields are equal'])
        else
            disp([fnamejoint{i} ' fields are not equal'])
        end
    end
    disp('Unique for data1:')
    disp(setdiff(fname1,fname2)')
    disp('Unique for data2:')
    disp(setdiff(fname2,fname1)')
end

Please note: the script above detects only limited number of “equality” cases. Due to a finite number of digits reserved for storing a number in memory (32 bits for double, 16 bit for single), the final result may depend on the computing path. For example, in theory, [math]1+a-1-a\equiv 0[/math]. In computer reality this is not always the case. Computers cannot keep simultaneously information about very large and very small parts of the same number. Thus, it can get rid of the small part if needed. This problem has a special name “rounding error”. If we compare [math]1/3+1-1[/math] and [math]1/3[/math], we obtain slightly different results:

>> a=1/3;
>> disp(a-a)
     0
>> disp(a+1-1-a)
  -5.5511e-17
>> disp(a+10-10-a)
   6.1062e-16
>>disp(isequal(a,a+1-1))
   0

The difference is very often negligible. However, a standard comparison using isequal does not work. As a result, if you want to compare two numbers/vectors/arrays, you have to specify your tolerance level (precision) and compare not vectors or matrices, but rather a measure of the distance between the two:

%function comparing two vectors with predefined tolerance level tol:
function out=myisequal(a,b,tol)
if mean(abs(a-b))>tol %checks whether the average absolute difference between the two vectors is larger than the tolerance level.
    out=0; %if "Yes", then these two vectors are different given the current tolerance level
else
    out=1; %if "No", then these two vectors are equal given the current tolerance level
end

Another command that generates a (possibly empty) array of structures is dir(file_mask). To obtain a list of all mat files in the current directory, you have to type:

>>   matfiles=dir('*.mat')
matfiles =
1004x1 struct array with fields:
    name
    date
    bytes
    isdir
    datenum

Then you can load all these files to memory (if you have enough of RAM). If all mat files in the directory have the same variables, you can load all of them by constructing a vector of structures. However, if the number of variables varies from file to file, using cell arrays would be a better idea (for details, see Section [cell]).

Cell Arrays[cell]

The least structured variable in MATLAB is cell array. You can store a sequence of any data objects in it. You may want to think about cell arrays as a structure where each field is coded by one number if it is one-dimensional or some set of numbers if it is multidimensional. Since everything is coded with numbers, it is easier to use cell arrays inside loops. However, since sequence of numbers is not as informative as structure field names, it is harder to follow. Cell arrays are defined using curly brackets:

>> c={'test',rand(10),1:10 }
c =
    'test'    [10x10 double]    [1x10 double]

The same result can be achieved in three steps:

>>  c{1}='test';
>>  c{2}=rand(10);
>>  c{3}=1:10;
>>  c
c =
    'test'    [10x10 double]    [1x10 double]

To refer to the second element of c(3), the following syntax has to be employed:

>> c{3}(2)
ans =
      2

Important:

There is a very important difference between c(3) and c{3}. The former refers to [math]1\times 1[/math] cell array, while the second refers to its value.

>>  mean(c(3));
Undefined function 'sum' for input arguments
of type 'cell'.
Error in mean (line 28)
  y = sum(x)/size(x,dim);

Since math functions are not defined on cell variables, while

>> mean(c{3})
ans =
    5.5000

works as expected. By the same logic, the assignment below

  c(4:5)={'ostrich', randn(50)};

works by extending cell array c to [math]5\times 1[/math], while

  c{4:5}={'ostrich', randn(50)};

generates an error. Also an error is generated by

  c(6)=randn(4);

as we are trying to assign a double array randn(4) to a cell element c(6). The correct assignment is

  c(6)={randn(4)};

Curly brackets convert a double array to a cell element, and you can assign a cell element to another cell element.

Now, equipped with this information, we can load all mat files in current directory in MATLAB using cell arrays:

  fname=dir('*.mat');
  for i=1:length(fname)
    data{i}=load(fname(1).name);
  end

The same goal can be achieved using structure:

  fname=dir('*.mat');
  for i=1:length(fname)
    curr_field=['f' fname(i).name(1:end-4)];%Filename is selected, .mat is dropped. The leading 'f' stands for a file. Please note that the code might not work for some file names
    data.(curr_field)=load(fname(i).name);
  end

It is possible to convert structures to cell arrays and vice versa. Also, it is possible to apply a function to each cell of a cell array

• c = struct2cell(s) converts a structure s to a cell array c, fields are converted to cells using the same order as in structure s

• s = cell2struct(c,fields) converts a cell array c to a structure array s with fieldnames defined in fields

    >> s=struct('f1',rand(10),'f2','MSFT') %an alternative way to create a structure
  s =
      f1: [10x10 double]
      f2: 'MSFT'
  >> fldnm=fieldnames(s)' %recording fieldnames of structure s
  fldnm =
      'f1'    'f2'
  >> c=struct2cell(s)'  %converting structure s to cell array c
  c =
      [10x10 double]    'MSFT'
  >> s2=cell2struct(c',fldnm) %converting cell array c to structure s2 using fldnm which was recorded earlier. For some reason MATLAB prefers column-vector c
  s2 =
      f1: [10x10 double]
      f2: 'MSFT'

• cellfun(@function_name,cell1,cell2,...,celln,options) evaluates a function
  function_name picking every first element of cell arrays cell1,cell2,...,celln, every second element of cell arrays cell1,cell2,...,celln, etc.

  >> c1={rand(10,10,10),rand(10,10,10),rand(10,10,10)};
  >> c2={1,2,3};
  >> average=cellfun(@mean,c1,c2, 'UniformOutput',0)
  average =
      [1x10x10 double]    [10x1x10 double]    [10x10 double]

  Without ’UniformOutput’,0 MATLAB tries to construct a vector with three elements and fails for obvious reasons.