cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Website generation on multiple core/cpu

Senior Member

Website generation on multiple core/cpu

hello,

 

Is there any way to generate website on multiplie core/cpu in order to improvement generation time?

 

So far only one core is mobilized and it can significantly improve the generation speed if the remaining cores are used.

 

 

thank you,

 

 

 

7 Replies
MEGA

Re: Website generation on multiple core/cpu

Hello

 

HOPEX does not enable this kind (parallel processing) of static web site generation

 

To reduce generation time, consider various indications

 

Deployment

  • Increase CPU frequency/ram of the machine where generation runs
  • Improve performances of network connection between machine where generation runs and database server (for RDBMS storage)
  • Avoid to cross network to save html pages (set target generation folder on local disk)

Web site design

 

 

 

Jerome
Contributor

Re: Website generation on multiple core/cpu

If we can request features I would really like if the website generation is actually updated to handle at least couple of threads, because that will decrease the generation time tremendously. Especially considering that we can’t make delta generation.

 

MEGA

Re: Website generation on multiple core/cpu

Hello Ki6i

 

As far as I know, there is no project to enable static web site generation with multiple threads or multiple cores.

The current bottleneck is the limit of a windows 32 process around 2.5 GB ram with recent operating systems.

 

An error handling mechanism exists. Consider to use it for incremental generation
Jerome
Contributor

Re: Website generation on multiple core/cpu

I meant delta in sense that it should generate only the objects that were modified since the last generation, because otherwise we endup generating every day 50 000+ pages and most often only 1000-2000 of these objects were actually updated.

 

Unfortunatelly there is no way to identify this objects in mega because the Modification Date is updated only on Attribute change and not on Association connect/disconnect change. Furthermore, the reports are clickable only if all the objects that are part of them are part of the scope of the generation.

 

 

MEGA

Re: Website generation on multiple core/cpu

Hello K6i6

 

In the mechanism described in the above KB (http://community.mega.com/t5/custom/page/page-id/m​ega-kb-solution?sid=50120000000mqmkAAA)

  • The first phase (to compute pages to be generated) is run from the beginning
  • The second phase is resumed from the last page generated (saved in a .ini file)

This mechanism does not rely of a persistent identification of 'new objects'

 

In general, if you need to identify objects that have changed of that are new, you can use the repository log (at repository level, not the private workspace log). This data material is used by features such as alert management or full search.

 

You could consider to update dynamically (with API script) a web site to control the list of objects to document (from the list identified in repository log)

However,you cannot control the scope of objects used to compute hyperlinks and indexes at generation

The hyperlinks and indexes would considers the 'complete scope' (previous list of objects + additional list of objects), only the additional scope.

 

 

 

Jerome
Occasional Contributor

Re: Website generation on multiple core/cpu

Bonjour Jerome,

are there any Changes to this Facts ? For us it is a Problem which we want to fix.

Cordialement Jens

Occasional Contributor

Re: Website generation on multiple core/cpu

Bonjour Jerome,

are there any Changes to this Facts ? For us it is a Problem which we want to fix.

Cordialement Jens