Current problems and intended solutions

There are several points in the language, where its logic would require solving fixed points of definitions. One such place is the descend statement. The obvious idea is, it should allow to use any variables and expressions. However, what would happen, if one file loaded by one descend statement would provide variable for computing the other one. This is forbidden by exception in the language currently.

Another such problem is in planned (and needed) feature for setting variables inside specified objects. But what if the variable specifying what objects are to be changed is influenced by some variable in other object, possibly set by this way? Or what if the object itself changes so much it returns different number of values after setting the value?

Need for such exceptions (as solving the fixed points is really not a way to go for a thing that is not really needed is an overkill) are ugly and the language should be redesigned in a way this is not needed or logically suggested by the rest of it.

Conditions are done with the if function, which is capable of doing anything needed in this area. However, writing again and again the same condition for each set variable is tedious. It would be more convenient to have a way to put more than one value into under a condition. However, such thing does not nicely fit into the syntax of the language and even less to current evaluation ways.

Variable name is not enough as an identifier of it, as isn’t filename of object. There may be more variables of the same name in different contexts and there may be more objects with the same filename, as they inherit other objects.

Therefore, storing values into cache makes a little problem. It is solved by storing everything, including the syntax trees. This has two problems. First is, if a source script file changes, nearly everything needs to be trashed, since it is heavily pointer bound and part of them is old, part is not know, where it belongs, since the pointers are used as IDs. Currently, remake restarts again if such situation is detected and does not load this part of cache. This is both ugly and unwanted, since it triggers recompilation of everything (all the objects are created again).

Another problem is it is not possible to write working eval, since it is impossible to check, if the input changed from last time.

Besides of these things, it is rather hard to live with such system and makes the binary cache unnecessarily complicated, which leads to large number of bugs.

It would be better to build everything every time and have some kind of identifiers surviving change of the source file. Then the value could be remembered.

Instead of marking which variable should be computed every time (which is not yet implemented), it would be easier and cleaner to mark which variables should be stored. The user would have better control over what is happening, the cache would be much smaller and lot of code would become unneeded.

Undefined values of variables act the same way as empty variables. This can be handy sometimes, but it is not really what you want when you make a typo in the name of function, parameter or somewhere else. Then you will spend several hours thinking, why does the function does not run, or it compiles only half of source files.

Undefined variables should be forbidden at most places. However, it should be possible to work with undefined variables. It is easy for user to set configuration options trough variables, but it makes sense for him to specify nothing when he wants the default one. It leads to undefined variables and it must be possible to work with them somehow.

It is hard to know when a variable stops being useful and can be freed. Memory pools solve this by freeing all at once. Some values are created as temporary, some as permanent and when a task is finished, everything temporary is freed.

This has two downsides. First, it brings more complexity to the code — there is need to mark variables, if they are permanent or temporary, if it is recognized that previously temporary value is needed permanently, it needs to be copied, there is need to pass the memory pools around, etc.

The second problem is, the memory gets freed much later than actually needed. The biggest occurrence is when the TARGETS variable is computed, since all the objects are created. Many of these variables created temporary are needed only for evaluating local expressions (parameters of functions, for example) and stop being needed right away. However, they stay in memory until the whole variable is evaluated. The remakes build system eats around 20 pages of memory for this purpose and returns them all at once, but the memory could be reused. This leads to unneeded memory consumption. However, this is not as bad as it looks, since the following compilation is expected to need memory too and the memory is returned to OS before it starts.

There are two possible solutions to this problems, but both require other language than C. First one is using garbage collector. But garbage collectors are in languages like Java or Haskell. They need runtime and they are slow. Besides, the garbage collector has quite some overhead, so the gain here is dubious, even without considering the need to rewrite the code.

The other one is using reference counters. Both memory and CPU overhead is small and remakes data structures do not have circular references (with few exceptions that would need to be solved manually or eliminated). Using reference counters in C brings much more code, and needs to be considered while writing it, which in turn leads to higher bug probability. But it would be possible to implement them in C++. And C++ is similar to C, so much of the code could be reused. However, it is still complete rewrite of the code.

When a compilation fails, cache is not stored. It can not be, since the values of variables are not in persistent state. It however means that when the bug in code is fixed, remake will compile again even the objects that succeeded last time.

This needs to be solved by storing much less information to cache and do some kind of transaction mechanism — all the variables of the objects are pushed to cache when the object succeeds and the previous version is used, if it does not.