Input normalization as separate layer in CNTK with C#

In the previous post, we have seen how to calculate some of basis parameters of descriptive statistics, as well as how to normalize data by calculating  mean and standard deviation. In this blog post we are going to implement data normalization as regular neural network layer, which can simplify the training process and data preparation.

What is Data normalization?

Simple said, data normalization is set of tasks which transform values of any feature in a data set into predefined number range. Usually this range is [-1,1] , [0,1] or some other specific ranges. Data normalization plays very important role in ML, since it can dramatically improve the training process, and simplify settings of network parameters.

There are two main types of data normalization:
– MinMax normalization – which transforms all values into range of [0,1],
– Gauss Normalization or Z score normalization, which transforms the value in such a way that the average value is zero, and std is 1.

Beside those types there are plenty of other methods which can be used. Usually those two are used when the size of the data set is known, otherwise we should use some of the other methods, like log scaling, dividing every value with some constant, etc. But why data need to be normalized? This is essential question in ML, and the simplest answer is to provide the equal influence to all features to change the output label. More about data normalization and scaling can be found on this link.

In this blog post we are going to implement CNTK neural network which contain a “Normalization layer” between input and first hidden layer. The schematic picture of the network looks like the following image:

As can be observed, the Normalization layer is placed between input and first hidden layer. Also the Normalization layer contains the same neurons as input layer and produced the  output with the same dimension as the input layer.

In order to implement Normalization layer the following requirements must be met:

• calculate average  $\mu$ and standard deviation $\sigma$ in training data set as well find maximum and minimum value of each feature.
• this must be done prior to neural network model creation, since we need those values in the normalization layer.
• within network model creation, the normalization layer should be define after input layer is defined.

Calculation of mean and standard deviation for training data set

Before network creation, we should prepare mean and standard deviation parameters which will be used in the Normalization layer as constants. Hopefully, the CNTK has the static method in the Minibatch source class for this purpose “MinibatchSource.ComputeInputPerDimMeansAndInvStdDevs”. The method takes the whole training data set defined in the minibatch and calculate the parameters.

//calculate mean and std for the minibatchsource
// prepare the training data
var d = new DictionaryNDArrayView, NDArrayView>>();
using (var mbs = MinibatchSource.TextFormatMinibatchSource(
trainingDataPath , streamConfig, MinibatchSource.FullDataSweep,false))
{
//compute mean and standard deviation of the population for inputs variables
MinibatchSource.ComputeInputPerDimMeansAndInvStdDevs(mbs, d, device);

}

Now that we have average and std values for each feature, we can create network with normalization layer. In this example we define simple feed forward NN with 1 input, 1 normalization, 1 hidden and 1 output layer.

private static Function createFFModelWithNormalizationLayer(Variable feature, int hiddenDim,int outputDim, Tuple avgStdConstants, DeviceDescriptor device)
{
//First the parameters initialization must be performed
var glorotInit = CNTKLib.GlorotUniformInitializer(
CNTKLib.DefaultParamInitScale,
CNTKLib.SentinelValueForInferParamInitRank,
CNTKLib.SentinelValueForInferParamInitRank, 1);

//*******Input layer is indicated as feature
var inputLayer = feature;

//*******Normalization layer
var mean = new Constant(avgStdConstants.Item1, "mean");
var std = new Constant(avgStdConstants.Item2, "std");
var normalizedLayer = CNTKLib.PerDimMeanVarianceNormalize(inputLayer, mean, std);

//*****hidden layer creation
//shape of one hidden layer should be inputDim x neuronCount
var shape = new int[] { hiddenDim, 4 };
var weightParam = new Parameter(shape, DataType.Float, glorotInit, device, "wh");
var biasParam = new Parameter(new NDShape(1, hiddenDim), 0, device, "bh");
var hidLay = CNTKLib.Times(weightParam, normalizedLayer) + biasParam;
var hidLayerAct = CNTKLib.ReLU(hidLay);

//******Output layer creation
//the last action is creation of the output layer
var shapeOut = new int[] { 3, hiddenDim };
var wParamOut = new Parameter(shapeOut, DataType.Float, glorotInit, device, "wo");
var bParamOut = new Parameter(new NDShape(1, 3), 0, device, "bo");
var outLay = CNTKLib.Times(wParamOut, hidLayerAct) + bParamOut;
return outLay;
}

Complete Source Code Example

The whole source code about this example is listed below. The example show how to normalize input feature for Iris famous data set. Notice that when using such way of data normalization, we don’t need to handle  normalization for validation or testing data sets, because data normalization  is part of the network model.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using CNTK;
namespace NormalizationLayerDemo
{
class Program
{
static string trainingDataPath = "./data/iris_training.txt";
static string validationDataPath = "./data/iris_validation.txt";
static void Main(string[] args)
{
DeviceDescriptor device = DeviceDescriptor.UseDefaultDevice();

//stream configuration to distinct features and labels in the file
var streamConfig = new StreamConfiguration[]
{
new StreamConfiguration("feature", 4),
new StreamConfiguration("flower", 3)
};

// build a NN model
//define input and output variable and connecting to the stream configuration
var feature = Variable.InputVariable(new NDShape(1, 4), DataType.Float, "feature");
var label = Variable.InputVariable(new NDShape(1, 3), DataType.Float, "flower");

//calculate mean and std for the minibatchsource
// prepare the training data
var d = new Dictionary();
using (var mbs = MinibatchSource.TextFormatMinibatchSource(
trainingDataPath , streamConfig, MinibatchSource.FullDataSweep,false))
{
//compute mean and standard deviation of the population for inputs variables
MinibatchSource.ComputeInputPerDimMeansAndInvStdDevs(mbs, d, device);

}

//Build simple Feed Froward Neural Network with normalization layer
var ffnn_model = createFFModelWithNormalizationLayer(feature,5,3,d.ElementAt(0).Value, device);

//Loss and error functions definition
var trainingLoss = CNTKLib.CrossEntropyWithSoftmax(new Variable(ffnn_model), label, "lossFunction");
var classError = CNTKLib.ClassificationError(new Variable(ffnn_model), label, "classificationError");

// set learning rate for the network
var learningRatePerSample = new TrainingParameterScheduleDouble(0.01, 1);

//define learners for the NN model
var ll = Learner.SGDLearner(ffnn_model.Parameters(), learningRatePerSample);

//define trainer based on model, loss and error functions , and SGD learner
var trainer = Trainer.CreateTrainer(ffnn_model, trainingLoss, classError, new Learner[] { ll });

//Preparation for the iterative learning process

// create minibatch for training
var mbsTraining = MinibatchSource.TextFormatMinibatchSource(trainingDataPath, streamConfig, MinibatchSource.InfinitelyRepeat, true);

int epoch = 1;
while (epoch  a.sweepEnd))
{
reportTrainingProgress(feature, label, streamConfig, trainer, epoch, device);
epoch++;
}
}
}

private static void reportTrainingProgress(Variable feature, Variable label, StreamConfiguration[] streamConfig,  Trainer trainer, int epoch, DeviceDescriptor device)
{
// create minibatch for training
var mbsTrain = MinibatchSource.TextFormatMinibatchSource(trainingDataPath, streamConfig, MinibatchSource.FullDataSweep, false);
var trainD = mbsTrain.GetNextMinibatch(int.MaxValue, device);
//
var a1 = new UnorderedMapVariableMinibatchData();
var trainEvaluation = trainer.TestMinibatch(a1);

// create minibatch for validation
var mbsVal = MinibatchSource.TextFormatMinibatchSource(validationDataPath, streamConfig, MinibatchSource.FullDataSweep, false);
var valD = mbsVal.GetNextMinibatch(int.MaxValue, device);

//
var a2 = new UnorderedMapVariableMinibatchData();
var valEvaluation = trainer.TestMinibatch(a2);

Console.WriteLine($"Epoch={epoch}, Train Error={trainEvaluation}, Validation Error={valEvaluation}"); } private static Function createFFModelWithNormalizationLayer(Variable feature, int hiddenDim,int outputDim, Tuple avgStdConstants, DeviceDescriptor device) { //First the parameters initialization must be performed var glorotInit = CNTKLib.GlorotUniformInitializer( CNTKLib.DefaultParamInitScale, CNTKLib.SentinelValueForInferParamInitRank, CNTKLib.SentinelValueForInferParamInitRank, 1); //*******Input layer is indicated as feature var inputLayer = feature; //*******Normalization layer var mean = new Constant(avgStdConstants.Item1, "mean"); var std = new Constant(avgStdConstants.Item2, "std"); var normalizedLayer = CNTKLib.PerDimMeanVarianceNormalize(inputLayer, mean, std); //*****hidden layer creation //shape of one hidden layer should be inputDim x neuronCount var shape = new int[] { hiddenDim, 4 }; var weightParam = new Parameter(shape, DataType.Float, glorotInit, device, "wh"); var biasParam = new Parameter(new NDShape(1, hiddenDim), 0, device, "bh"); var hidLay = CNTKLib.Times(weightParam, normalizedLayer) + biasParam; var hidLayerAct = CNTKLib.ReLU(hidLay); //******Output layer creation //the last action is creation of the output layer var shapeOut = new int[] { 3, hiddenDim }; var wParamOut = new Parameter(shapeOut, DataType.Float, glorotInit, device, "wo"); var bParamOut = new Parameter(new NDShape(1, 3), 0, device, "bo"); var outLay = CNTKLib.Times(wParamOut, hidLayerAct) + bParamOut; return outLay; } } } The output window should looks like: The data set files used in the example can be downloaded from here, and full source code demo from here. Advertisements Descriptive statistics and data normalization with CNTK and C# As you probably know CNTK is Microsoft Cognitive Toolkit for deep learning. It is open source library which is used by various Microsoft products. Also the CNTK is powerful library for developing custom ML solutions from various fields with different platforms and languages. What is also so powerful in the CNTK is the way of the implementation. In fact the library is implemented as series of computation graphs, which is fully elaborated into the sequence of steps performed in a deep neural network training. Each CNTK compute graph is created with set of nodes where each node represents numerical (mathematical) operation. The edges between nodes in the graph represent data flow between operations. Such a representation allows CNTK to schedule computation on the underlying hardware GPU or CPU. The CNTK can dynamically analyze the graphs in order to to optimize both latency and efficient use of resources. The most powerful part of this is the fact thet the CNTK can calculate derivation of any constructed set of operations, which can be used for efficient learning process of the network parameters. The flowing image shows the core architecture of the CNTK. On the other hand, any operation can be executed on CPU or GPU with minimal code changes. In fact we can implement method which can automatically takes GPU computation if available. The CNTK is the first .NET library which provide .NET developers to develop GPU aware .NET applications. What this exactly mean is that with this powerful library you can develop complex math computation directly to GPU in .NET using C#, which currently is not possible when using standard .NET library. For this blog post I will show how to calculate some of basic statistics operations on data set. Say we have data set with 4 columns (features) and 20 rows (samples). The C# implementation of this 2D array is show on the following code snippet: static float[][] mData = new float[][] { new float[] { 5.1f, 3.5f, 1.4f, 0.2f}, new float[] { 4.9f, 3.0f, 1.4f, 0.2f}, new float[] { 4.7f, 3.2f, 1.3f, 0.2f}, new float[] { 4.6f, 3.1f, 1.5f, 0.2f}, new float[] { 6.9f, 3.1f, 4.9f, 1.5f}, new float[] { 5.5f, 2.3f, 4.0f, 1.3f}, new float[] { 6.5f, 2.8f, 4.6f, 1.5f}, new float[] { 5.0f, 3.4f, 1.5f, 0.2f}, new float[] { 4.4f, 2.9f, 1.4f, 0.2f}, new float[] { 4.9f, 3.1f, 1.5f, 0.1f}, new float[] { 5.4f, 3.7f, 1.5f, 0.2f}, new float[] { 4.8f, 3.4f, 1.6f, 0.2f}, new float[] { 4.8f, 3.0f, 1.4f, 0.1f}, new float[] { 4.3f, 3.0f, 1.1f, 0.1f}, new float[] { 6.5f, 3.0f, 5.8f, 2.2f}, new float[] { 7.6f, 3.0f, 6.6f, 2.1f}, new float[] { 4.9f, 2.5f, 4.5f, 1.7f}, new float[] { 7.3f, 2.9f, 6.3f, 1.8f}, new float[] { 5.7f, 3.8f, 1.7f, 0.3f}, new float[] { 5.1f, 3.8f, 1.5f, 0.3f},}; If you want to play with CNTK and math calculation you need some knowledge from Calculus, as well as vectors, matrix and tensors. Also in CNTK any operation is performed as matrix operation, which may simplify the calculation process for you. In standard way, you have to deal with multidimensional arrays during calculations. As my knowledge currently there is no .NET library which can perform math operation on GPU, which constrains the .NET platform for implementation of high performance applications. If we want to compute average value, and standard deviation for each column, we can do that with CNTK very easy way. Once we compute those values we can used them for normalizing the data set by computing standard score (Gauss Standardization). The Gauss standardization is calculated by the flowing term: $nValue= \frac{X-\nu}{\sigma}$, where X- is column values, $\nu$ – column mean, and $\sigma$– standard deviation of the column. For this example we are going to perform three statistic operations,and the CNTK automatically provides us with ability to compute those values on GPU. This is very important in case you have data set with millions of rows, and computation can be performed in few milliseconds. Any computation process in CNTK can be achieved in several steps: 1. Read data from external source or in-memory data, 2. Define Value and Variable objects. 3. Define Function for the calculation 4. Perform Evaluation of the function by passing the Variable and Value objects 5. Retrieve the result of the calculation and show the result. All above steps are implemented in the following implementation: using System; using System.Collections.Generic; using System.Diagnostics; using System.Linq; using System.Text; using System.Threading.Tasks; using CNTK; namespace DataNormalizationWithCNTK { class Program { static float[][] mData = new float[][] { new float[] { 5.1f, 3.5f, 1.4f, 0.2f}, new float[] { 4.9f, 3.0f, 1.4f, 0.2f}, new float[] { 4.7f, 3.2f, 1.3f, 0.2f}, new float[] { 4.6f, 3.1f, 1.5f, 0.2f}, new float[] { 6.9f, 3.1f, 4.9f, 1.5f}, new float[] { 5.5f, 2.3f, 4.0f, 1.3f}, new float[] { 6.5f, 2.8f, 4.6f, 1.5f}, new float[] { 5.0f, 3.4f, 1.5f, 0.2f}, new float[] { 4.4f, 2.9f, 1.4f, 0.2f}, new float[] { 4.9f, 3.1f, 1.5f, 0.1f}, new float[] { 5.4f, 3.7f, 1.5f, 0.2f}, new float[] { 4.8f, 3.4f, 1.6f, 0.2f}, new float[] { 4.8f, 3.0f, 1.4f, 0.1f}, new float[] { 4.3f, 3.0f, 1.1f, 0.1f}, new float[] { 6.5f, 3.0f, 5.8f, 2.2f}, new float[] { 7.6f, 3.0f, 6.6f, 2.1f}, new float[] { 4.9f, 2.5f, 4.5f, 1.7f}, new float[] { 7.3f, 2.9f, 6.3f, 1.8f}, new float[] { 5.7f, 3.8f, 1.7f, 0.3f}, new float[] { 5.1f, 3.8f, 1.5f, 0.3f},}; static void Main(string[] args) { //define device where the calculation will executes var device = DeviceDescriptor.UseDefaultDevice(); //print data to console Console.WriteLine($"X1,\tX2,\tX3,\tX4");
Console.WriteLine($"-----,\t-----,\t-----,\t-----"); foreach (var row in mData) { Console.WriteLine($"{row[0]},\t{row[1]},\t{row[2]},\t{row[3]}");
}
Console.WriteLine($"-----,\t-----,\t-----,\t-----"); //convert data into enumerable list var data = mData.ToEnumerable<IEnumerable<float>>(); //assign the values var vData = Value.CreateBatchOfSequences<float>(new int[] {4},data, device); //create variable to describe the data var features = Variable.InputVariable(vData.Shape, DataType.Float); //define mean function for the variable var mean = CNTKLib.ReduceMean(features, new Axis(2));//Axis(2)- means calculate mean along the third axes which represent 4 features //map variables and data var inputDataMap = new Dictionary<Variable, Value>() { { features, vData } }; var meanDataMap = new Dictionary<Variable, Value>() { { mean, null } }; //mean calculation mean.Evaluate(inputDataMap,meanDataMap,device); //get result var meanValues = meanDataMap[mean].GetDenseData<float>(mean); Console.WriteLine($"");
Console.WriteLine($"Average values for each features x1={meanValues[0][0]},x2={meanValues[0][1]},x3={meanValues[0][2]},x4={meanValues[0][3]}"); //Calculation of standard deviation var std = calculateStd(features); var stdDataMap = new Dictionary<Variable, Value>() { { std, null } }; //mean calculation std.Evaluate(inputDataMap, stdDataMap, device); //get result var stdValues = stdDataMap[std].GetDenseData<float>(std); Console.WriteLine($"");
Console.WriteLine($"STD of features x1={stdValues[0][0]},x2={stdValues[0][1]},x3={stdValues[0][2]},x4={stdValues[0][3]}"); //Once we have mean and std we can calculate Standardized values for the data var gaussNormalization = CNTKLib.ElementDivide(CNTKLib.Minus(features, mean), std); var gaussDataMap = new Dictionary<Variable, Value>() { { gaussNormalization, null } }; //mean calculation gaussNormalization.Evaluate(inputDataMap, gaussDataMap, device); //get result var normValues = gaussDataMap[gaussNormalization].GetDenseData<float>(gaussNormalization); //print data to console Console.WriteLine($"-------------------------------------------");
Console.WriteLine($"Normalized values for the above data set"); Console.WriteLine($"");
Console.WriteLine($"X1,\tX2,\tX3,\tX4"); Console.WriteLine($"-----,\t-----,\t-----,\t-----");
var row2 = normValues[0];
for (int j = 0; j < 80; j += 4)
{
Console.WriteLine($"{row2[j]},\t{row2[j + 1]},\t{row2[j + 2]},\t{row2[j + 3]}"); } Console.WriteLine($"-----,\t-----,\t-----,\t-----");
}

private static Function calculateStd(Variable features)
{
var mean = CNTKLib.ReduceMean(features,new Axis(2));
var remainder = CNTKLib.Minus(features, mean);
var squared = CNTKLib.Square(remainder);
//the last dimension indicate the number of samples
var n = new Constant(new NDShape(0), DataType.Float, features.Shape.Dimensions.Last()-1);
var elm = CNTKLib.ElementDivide(squared, n);
var sum = CNTKLib.ReduceSum(elm, new Axis(2));
var stdVal = CNTKLib.Sqrt(sum);
return stdVal;
}
}

public static class ArrayExtensions
{
public static IEnumerable<T> ToEnumerable<T>(this Array target)
{
foreach (var item in target)
yield return (T)item;
}
}
}

The output for the source code above should look like:

MicrosoftML package

Microsoft has released Microsoft R Server 9.0 (MRS9.0) with very interesting package called MicrosoftML. “Micrsooft ML” stands for Microsoft Machine Learning R package which you can use on R Server. R Server is commercial version of popular R Client distribution, which solves mayor problems when working with R. R Server contains set of cutting-edge technology to work with big data, as well as set of enhanced packages for parallelization and distributing computing.
MRS 9.0 is coming with “MicrosoftML” package which contains set of several Machine Learning algorithms developed in various Microsoft products in the last 10 years. You can combine the algorithms delivered in this package with pre-existing parallel external memory algorithms such as the RevoScaleR package as well as open source innovations such as CRAN R packages to deliver the best predictive analytic.
MicrosoftML package includes the following algorithms:

• Fast linear learner, with support for L1 and L2 regularization,
• Fast boosted decision tree,
• Fast random forest,
• Logistic regression, with support for L1 and L2 regularization,
• GPU-accelerated Deep Neural Networks (DNNs) with convolutions,
• Binary classification using a One-Class Support Vector Machine.

In order to fully use the power of MicrosoftML, and RevoScaleR you need to download MRS 9.0 from the MSDN or Visual Studio Dev Essentials subscription. Once the zip file is downloaded, unzip it, and run setup file.

The following required components  were missing when my installation is started.  Seems the MRS contains the latest .NET Core components, which is pretty cool:

After the prerequested components installed, the MRS installation process can start.

By clicking the Next button the Installation process starts:

Select the path where you want to install MRS, and press the Next button:

If everything went ok, the installation process is finished after less than minute, and the final dialog window appears:

By clicking the Finish button MRS is installed on you PC.

Run MRS 9.0 by using R Tool fo Visual Studio, RTVS

Now it is time to run some R code. YOu have two posibilities to run R code. The first option is that you use the R Studio proffesion tool for running R code. It is free and open source which you can download from rstudio.com. If you are MS Developer you usualy write the code in the Visual Studio. So you can download RTVS from this link and run R code from Visual Studio.

Now that you have right tool to run R code, we can start with setting the MRS environment.

First thing you should do is to point RTVS to use MRS 9.0 instead of curently using some other distribution. So open the Visual Studio, select R Tools->Edit Options

The Option dialog appears. Set the R Engine to point installation folder of the MRS. Since my installation location was on Program Files folder, the picture below show my installation path.

After you set the right installation folder , restart the Visual Studio:

When the Visual Studio is running, open R Open R Interactive window. You should have similar text if you set up MRS path correctly:

Select New Project from the File->New menu option.

Name it FirstRServerDemo and click Ok. Now you are ready to write first MRS R code:

In the next post we will continue exploration the MicrosoftML library package and new set of Machine Learning algorithms added in this latest version.

Details of my session at ATD12

Today, I gave session at Advanced Technology Day conference in Zagreb. It was very excited to see full room of people at the presentation, mostly developers from .NET world interesting in R and Data Science. This is good sign that the Data Science and the R are becoming more and more popular at daily basis. Most popularity for the R will bring R Tool for Visual Studio, which means the R language became member of the family of the Visual Studio.

For those who were asking about my slides and demo sample here is the information:

2. Source code of the demo is hosted at git hub at: http://github.com/bhrnjica/R-Workshop

See you next time!

Uvod u R jezik i mašinsko učenje- youtube lekcije

Do sada urađeno 7 kratkih lekcija u kojima sam pokusao objasniti osnove programskog jezika R i kako koristiti funkcije za statističku i analitičku obradu podataka. Kratkih uradaka biće još, ovo je samo početak. U narednom tekstu možete vidjeti sve do sada objavljene lekcije.

Visual Studio vNext – The New Installer

The new version of Visual Studio will come with dramatically new installer, which will allow that you install only stuff you need, without gigabytes of unnecessary never used components. Current version of Visual Studio which is Visual Studio 2015 Update 3 is coming with nearly 8GB installation file. This is to much for the installer, you need special condition when you want to download the installation file. I am doing it by night, when I am sleeping. In some condition the installation process takes an hour to install everything you have specified.

In the next version the installation process will be changed and if you want to see and feel how the future visual studio installer  will look like you can download the preview of the Visual Studio vnext code name  “Visual Studio 15” at this link.

If you try to install Visual Studio 15 preview 3, it will take less than 5 minutes, with very simple installer. In the next five pictures whole installation process is completed.

After you download the installer, run it and the following pictures will appear:

1. First picture is asking to confirm the installation process:

3. The next picture is the main picture which you can select what to install. The whole Visual Studio installer is devided in to the development groups:

1. Core Stuff of the Visual Studio- this component is required for all developer group
2. There are for now 4 installer groups: .NET, C++, Python, Game dev.
3. The more will come later.

4. After you select right developer group/groups installation process starts by pressing Install button.

5. After the installation process is completed, the following picture appear, which you only need to close by pressing the Close button at the right top edge of the window.

As we can see the next version of the Visual Studio will dramatically changed the installation process, offering new simple and effective installer.

Using external config files in .NET applications

The config file is place where common variables, database connection strings, web page settings and other common stuff are placed. The config file is also dynamic, so you can change the value of the variable in the config file  without compiling and deploying the .NET app. In multi tenancy environment config file can be complicate for deployment, because  for each tenant different value must be set for most of the defined variables. In such a situation you have to be careful to set right value for the right tenant.

One way of handling this is to hold separate config file for each tenant. But the problem can be variables which are the same for all tenants, and also the case where some variables can be omitted for certain tenant.

One of the solution for this can be defining external config files for only connection strings or appSettings variables, or any other custom config section. In this blog post, it will be presenting how to define connection strings as well as appSettings section in separate config file.

Lets say you have appSettings and connectionStrings config sections, similar like code below:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
</startup>

<connectionStrings>
<add name="SQLConnectionString01" connectionString="Data Source=sourcename01;Initial Catalog=cat01;Persist Security Info=True;Integrated Security=true;"/>
<add name="SQLConnectionString02" connectionString="Data Source=sourcename02;Initial Catalog=cat02;Persist Security Info=True;Integrated Security=true;"/>
</connectionStrings>

<appSettings>
<clear />
<!-- Here are list of appsettings -->
<add key="Var1" value="Var1 value from config01" />
<add key="Var2" value="Varn value from config01"/>
<add key="Var3" value="Var3 value from main config file"/>
</appSettings>

</configuration>

There are three appSetting keys Var1 , Var2 and Var3  and two connectionstrings in the app.config.

The config file above can be split in such a way that variables Var1 and Var2 be defined in separated file, but the Var3 can be remain in the main cofing file. Separate config file may be unique for each tenant.

Now the main config file looks like the following:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
</startup>

<connectionStrings configSource="config\connString01.config"/>

<appSettings file="config\config01.config">

<add key="Var3" value="Var3 value from main config file"/>
</appSettings>

</configuration>

In the Visual Studio Solution there is config folder in which we created two config files for appSettings section and two config files for Connectionstrings section, in case we have two separate environments for deployments.

The flowing code snippet shows the appSettings section implemented in the external file:

<appSettings file="appSettings.config">

<!-- Here are list of appsettings -->
<add key="Var1" value="Var1 value from config02" />
<!-- ... -->
<add key="Varn" value="Varn value from config02"/>
</appSettings>

The external config file for connection strings looks similar like the flowing:

The simple console application shows how to use this config variables in the code:

static void Main(string[] args)
{
var var1Value= ConfigurationManager.AppSettings["Var1"];
var var2Value = ConfigurationManager.AppSettings["Var2"];
var var3Value = ConfigurationManager.AppSettings["Var3"];
var conn1 = ConfigurationManager.ConnectionStrings["SQLConnectionString01"];
var conn2 = ConfigurationManager.ConnectionStrings["SQLConnectionString02"];

Console.WriteLine("Values from config01.config and connString01.config files");

Console.WriteLine("Var1={0}",var1Value);
Console.WriteLine("Var2={0}", var2Value);
Console.WriteLine("Var3={0}", var3Value);
Console.WriteLine("ConnStr01={0}", conn1);
Console.WriteLine("ConnStr01={0}", conn2);