Code Sample: PartitionCollectionTest


About 3 years ago, I posted an article about Too much to digest? Slice it up in chunks!, which worked, but was a rather complex way of slicing items into smaller bites.

Fruit

For a while now, there has been an alternative; the PartitionCollection<T> class in the Cuemon.Collections.Generic namespace. Depending on your .NET installation, there are also extension methods ToPartitionCollection that might come in handy.

The usage is simple, as illustrated here (i have kept the example as close to the original as possible):

using System.Diagnostics;  
using System.Linq;  
using Cuemon.Collections.Generic;  
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace Cuemon.Examples  
{
    [TestClass]
    public class PartitionCollectionTest
    {
        [TestMethod]
        public void ChunkTest()
        {
            var workload = 4096;
            var partitionSize = 255;
            var simulatedWorkload = Enumerable.Range(0, workload).ToPartitionCollection(partitionSize);

            Trace.WriteLine("Work to process: {0} items.".FormatWith(workload));
            Trace.WriteLine("Partitions for work: {0} runs.".FormatWith(simulatedWorkload.PartitionCount()));

            while (simulatedWorkload.HasPartitions)
            {
                var chunkedWorkload = simulatedWorkload.ToList();
                foreach (var work in chunkedWorkload)
                {
                    // process your work here
                }
                
                Trace.WriteLine("Remaining work to process: {0} items.".FormatWith(simulatedWorkload.Remaining));
            }
        }
    }
}
The result is well familiar:
Work to process: 4096 items.  
Partitions for work: 17 runs.  
Remaining work to process: 3841 items.  
Remaining work to process: 3586 items.  
Remaining work to process: 3331 items.  
Remaining work to process: 3076 items.  
Remaining work to process: 2821 items.  
Remaining work to process: 2566 items.  
Remaining work to process: 2311 items.  
Remaining work to process: 2056 items.  
Remaining work to process: 1801 items.  
Remaining work to process: 1546 items.  
Remaining work to process: 1291 items.  
Remaining work to process: 1036 items.  
Remaining work to process: 781 items.  
Remaining work to process: 526 items.  
Remaining work to process: 271 items.  
Remaining work to process: 16 items.  
Remaining work to process: 0 items.  

As always; I hope you found these small articles inspiring - and happy coding!

Too much to digest? Slice it up in chunks!


From time to time you will find yourself in the need of slicing up large chunks of bites into smaller chunks of consumable bites.

Since I have been unable to find such a method in the .NET framework, I decided to include one on the EnumerableUtility class found in the Cuemon.Collections.Generic namespace. The name for the method was carefully chosen to be Chunk, and consist of two overloads; both accepting an IEnumerable<T> where the chunk size is defaulted to 128 elements and the other with a custom size.

To see the Chunk method in action, have a look at Figure 1. For simplicity i set a chunk size to 255 and a workload count to 4096. This means that our workload consists of 4096 elements which we will process 255 elements at time. This leaves us with a workload of 17 runs in total.


[TestClass]
public class ChunkExample
{
    [TestMethod]
    public void TestChunk()
    {
        int chunkSize = 255;
        int workload = 4096;
        int expectedWorkloadIterations = (workload + chunkSize - 1 ) / chunkSize;
        IEnumerable<int> simulatedWorkload = new List<int>(EnumerableUtility.Range(0, workload));

        Debug.WriteLine("Work to process: {0} items.", workload);
        Debug.WriteLine("Expected iterations for work: {0} runs.", expectedWorkloadIterations);

        int actualWorkloadIterations = 0;
        while (((List<int>)simulatedWorkload).Count > 0)
        {
            IEnumerable<int> chunkedWorkload = EnumerableUtility.Chunk(ref simulatedWorkload, chunkSize);
            IEnumerator<int> chunkedWorkloadEnumerator = chunkedWorkload.GetEnumerator();
            int countOfChunkedElements = EnumerableUtility.Count(chunkedWorkload);

            Assert.IsTrue(countOfChunkedElements <= chunkSize);

            int enumeratorCount = 0;
            while (chunkedWorkloadEnumerator.MoveNext())
            {
                enumeratorCount++;
            }
            actualWorkloadIterations++;

            Assert.IsTrue(enumeratorCount <= chunkSize);

            Debug.WriteLine("Remaining work to process: {0} items.", ((List<int>)simulatedWorkload).Count);
        }

        Assert.IsTrue(expectedWorkloadIterations == actualWorkloadIterations);
        Debug.WriteLine("Actual iterations for work: {0} runs.", actualWorkloadIterations);
    }
}

Figure 1: A simple test method that utilizes the Chunk method found on the EnumerableUtility class.




When executed, the test method should produce this debug output:




Debug Trace:
Work to process: 4096 items.
Expected iterations for work: 17 runs.
Remaining work to process: 3841 items.
Remaining work to process: 3586 items.
Remaining work to process: 3331 items.
Remaining work to process: 3076 items.
Remaining work to process: 2821 items.
Remaining work to process: 2566 items.
Remaining work to process: 2311 items.
Remaining work to process: 2056 items.
Remaining work to process: 1801 items.
Remaining work to process: 1546 items.
Remaining work to process: 1291 items.
Remaining work to process: 1036 items.
Remaining work to process: 781 items.
Remaining work to process: 526 items.
Remaining work to process: 271 items.
Remaining work to process: 16 items.
Remaining work to process: 0 items.
Actual iterations for work: 17 runs.

Figure 2: The output of the test in Figure 1.



I hope you liked this little peek into one of the many day-to-day helper methods found through out the Cuemon assembly family. Happy coding!