Memory at each node on a NUMA machine using libnuma (linux only)

For the past few days I have been occupied with trying to understand how NUMA works and what are related implications and challenges for system design. Linux supports more than a few memory location policies on a NUMA supported machine and a corresponding user space library to deal with these policies, migration of memory, thread location etc among other things. The library for most part seems to get information from the files in /proc/self and in /sys/devices/system/node. The latter location in the sys filesystem has a multitude of information pertaining to the memory on the system.
Here is a simple program I wrote toady which prints the memory located at each "defined" memory node on the NUMA system, this program uses the numa library libnuma. It is a rather trivial piece of code. My next task at hand (once I am done with the semester finals) would be to play around with my parallel implementation of QuickSort using pthreads and see if I can improve performance by manually playing with locality of memory and thread execution.

I did hit upon one important piece of information which I did not know before, that my laptop (DELL E6410) is NUMA compatible but has a single memory node with all the 8 Gigs of memory. This might as well be the reason why the parallel QuickSort worked faster on my laptop than on some of the department machines with distributed memory banks and faster truly multithreaded cores.

EDIT :

if you have the numactl package installed, you can also list numa related hardware information using the following command :

numactl --hardware

looking at the information you can tell that it is the same as what you would see under the various entries in the /sys/devices/system/node directory.

Converting a space delimited sentence into a sequence of null terminated strings

The title says it all, you are given a string which consists of words separated by spaces, here is a snippet of code for converting the same into a sequence of null terminated strings and then later printing them. Returning the indices in the original string where the words begin is the key here :




Printing the strings is pretty trivial as :

Quick code, picked up the hack from a certain shell command parsing code in Xinu.

Simple shared memory using mmap()

I was checking out some examples in the Linux Programming Interface. The idea of using shared memory bits between the children and the parent process for synchronization looked like a simple yet powerful idea, here is an example program that does exactly that :


Note : to map allocate the memory from the virtual memory space of the proccess, you have to use MAP_ANONYMOUS.

Checking fragmentation - part II

You should read the pre-cursor to this post here if you want to make any sense out of what I am going to write here. Anyway, I hacked the previous program further, being a person who always craves for some kind of visual representation, I decided to print the blocks of the file to the screen using a certain symbol for each block. Now this was fine for small files, but to really see some fragmentation happening you have to look at files in the order of a few hundreds of MB in size, and for such huge files, printing a symbol for each block will clutter the screen. So I decided to make the program a little more intelligent, it will instead print the blocks to the screen using the following key :
@ represents 10000 contiguous blocks on the disk
#represents 1000 contiguous blocks on the disk
$ represents 100 contiguous blocks on the disk
& represents 10 contiguous blocks on the disk
* represents 1  block on the disk
so now I print the blocks that have been used to allocate for the file and the blocks that do not belong to the file but which fall in between. The blocks which belong to the file are printed in blue while the ones which don't are printed in red.
I was searching for a highly fragmented file on my 1 TB external HD and found an iso image of about 1.2 GB which was fragmented at approx 200 places (thanks to ntfs on the external HD), so I ran the program on this file once and then copied the file to my laptop's HD(which has ext3 btw) and ran the program again, here are the results, the excessive fragmentation is visible in the first case:

tomorrow, I am going to try to create an image with the block map for a file, I will try using the gd library for this. More results tomorrow :). BTW, you can get the program from here.

Do you think it is stored together ? (checking fragmentation)

The title might be quite misleading, and hence I have put the real meaning in the brackets. The post is about using ioctl with the operation FIBMAP to check the block numbers allocated to a file in a sequential order. Whenever there is a jump in the block numbers, we know that there is some fragmentation in the way the file has been allocated on the disk. Looking at the code, it is pretty obvious that I have not written it (it is obvious from the usage of assert ;)), a little googling about FIBMAP landed me at this code. All I have done is modify the code to print only those block numbers where there appears to be fragmentation happening.The code is :




note : the FIBMAP operation is privileged, hence you will have to run the program once compiled as root. I am also adding the results I got for a test run on a file about 702 MB in size.

My next task would be to fill up most of the free space in the filesystem and then create some large files and check out the fragmentation for those files. For more information on fragmentation you could check out the useful information here.

using getopt : is it worth the effort ?

I was recently going through the GNU getopt library for parsing command line options. On a *nix system with online man pages installed, you should be able to read the manual pages for the getopt C library functions by doing something like
man 3 getopt
Now what I am left wondering about is, whether the complexity of using the library functions when the number of arguments your program requires is one or two, worth it ? Whenever I have had to implement similar functionality into my program all I have done is use a while loop over argv until I found all the required arguments. Something like this :
.
.
.
i = 0;
num_args_found = 0;
do{
 if (argv[i] == '-'){
           /*check for all valid options */
           /*if invalid character found then skip and continue */
 }
 i++;
 num_args_found++;
}while (i < argc);
if (num_args_found < requried_num_of_args){
 printf ("The requried arguments were not found\n");
 exit (0);
}
/*rest of the program here */.
.
.
though I am sure this is the best of the approaches, and even I have sometimes used a better one or a modified version of the above code, like to show exactly which arguments were missing. But the point of discussion here is that, when the expected number of arguments is less than three or four, the complexity of using a library function is much more than that for writing some custom code as above.