Thread: [qooxdoo-commit] SF.net SVN: qooxdoo: [10573] branches/legacy_0_7_x/qooxdoo/frontend/ framework/too
Brought to you by:
ecker,
martinwittemann
From: <wp...@us...> - 2007-10-15 10:52:38
|
Revision: 10573 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10573&view=rev Author: wpbasti Date: 2007-10-15 03:52:36 -0700 (Mon, 15 Oct 2007) Log Message: ----------- Backported generator2 to legacy... For the moment this will be the development version which will be ported again to the trunk afterwards. Added Paths: ----------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Added: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py (rev 0) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-15 10:52:36 UTC (rev 10573) @@ -0,0 +1,1877 @@ +#!/usr/bin/env python +################################################################################ +# +# qooxdoo - the new era of web development +# +# http://qooxdoo.org +# +# Copyright: +# 2006-2007 1&1 Internet AG, Germany, http://www.1and1.org +# +# License: +# LGPL: http://www.gnu.org/licenses/lgpl.html +# EPL: http://www.eclipse.org/org/documents/epl-v10.php +# See the LICENSE file in the project's top-level directory for details. +# +# Authors: +# * Sebastian Werner (wpbasti) +# +################################################################################ + +""" +Introduction +====================== +Replacement for old generator +Currently includes features of the old modules "generator" and "loader" + +Overview +====================== +* Load project configuration from JSON(-like) data file +* Each configuration can define multiple so named jobs +* Each job defines one action with all configuration +* A job can extend any other job and finetune the configuration +* Each execution of the generator can execute multiple of these jobs at once + +* The system supports simple include/exclude lists +* The smart mode (default) includes the defined classes and their dependencies +and excludes the defined classes and dependencies but does not break remaining +included features. +* Each generated script (named package here) contains the compiled JavaScript data +* It is possible to generate multiple variant combinations +* This means that a single job execution can create multiple files at once +* Variants are combineable and all possible combinations are automatically created. +For example: gecko+debug, mshtml+debug, gecko+nodebug, mshtml+nodebug + +* A further method to work with is the declaration of so named parts +* Each part defines a part of the application which you want to load separately +* A part could be of visual or logical nature +* Each part may result into multiple packages (script files) +* The number of packages could be exponential to the number of views +but through the optimization this is often not the case +* You can automatically collapse the important views. Such an important +view may be the initial application class (application layout frame) or +the splashscreen. Collapsing reduces the number of packages for the +defined views. However collapsing badly influences the fine-grained nature +of the package system and should be ommitted for non-initial views normally. +* Further optimization includes support for auto-merging small packages. +The relevant size to decide if a package is too small is the token size which +is defined by the author of the job. The system calculates the token size of +each package and tries to merge packages automatically. + +Internals +====================== +* All merges happen from right to left when the package list is sorted by priority. +The main theory is that a package which is used by multiple views must have the dependencies +solved by both of them. So the merge will always happen into the next common package of +both views from the current position to the left side. + +* There are some utility method which + +* The following global variables exist: + * classes{Dict}: All classes of the present class path configuration. Each entry + contains information regarding the path, the encoding, the class path and stuff + * modules{Dict}: All known modules from all available classes. Each entry contains + the classes of the current module + * verbose{Boolean}: If verbose mode is enabled + * quiet{Boolean}: If quiet mode is enabled + +* All cache data is automatically stored into framework/tool/.cache. The path is automatically + detected through the location of the generator script. +""" + + +import sys, re, os, optparse, math, cPickle, copy, sets, zlib + +# reconfigure path to import own modules from modules subfolder +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])), "modules")) + +import config, tokenizer, tree, treegenerator, treeutil, optparseext, filetool +import compiler, textutil, mapper +import variantoptimizer +import variableoptimizer, stringoptimizer, basecalloptimizer, privateoptimizer +import api +import simplejson + + + +###################################################################### +# MAIN CONTENT +###################################################################### + +def main(): + print "============================================================================" + print " INITIALIZATION" + print "============================================================================" + + parser = optparse.OptionParser(option_class=optparseext.ExtendAction) + + parser.add_option("-c", "--config", dest="config", metavar="FILENAME", help="Configuration file") + parser.add_option("-j", "--jobs", action="extend", dest="jobs", metavar="DIRECTORY", type="string", default=[], help="Selected jobs") + parser.add_option("-q", "--quiet", action="store_true", dest="quiet", default=False, help="Quiet output mode (Extra quiet).") + parser.add_option("-v", "--verbose", action="store_true", dest="verbose", default=False, help="Verbose output mode (Extra verbose).") + + if len(sys.argv[1:]) == 0: + basename = os.path.basename(sys.argv[0]) + print "usage: %s [options]" % basename + print "Try '%s -h' or '%s --help' to show the help message." % (basename, basename) + sys.exit(1) + + (options, args) = parser.parse_args(sys.argv[1:]) + + process(options) + + +def process(options): + global verbose + global quiet + + verbose = options.verbose + quiet = options.quiet + + if verbose: + quiet = False + + print ">>> Processing..." + if not quiet: + print " - Configuration: %s" % options.config + print " - Jobs: %s" % ", ".join(options.jobs) + + config = simplejson.loads(filetool.read(options.config)) + resolve(config, options.jobs) + + for job in options.jobs: + execute(job, config[job]) + + +def resolve(config, jobs): + print ">>> Resolving jobs..." + for job in jobs: + resolveEntry(config, job) + + +def resolveEntry(config, job): + global quiet + + if not config.has_key(job): + print " - No such job: %s" % job + sys.exit(1) + + data = config[job] + + if data.has_key("resolved"): + return + + #print " - Processing: %s" % job + + if data.has_key("extend"): + includes = data["extend"] + + for entry in includes: + resolveEntry(config, entry) + mergeEntry(config[job], config[entry]) + + data["resolved"] = True + + +def mergeEntry(target, source): + for key in source: + if not target.has_key(key): + target[key] = source[key] + + +def execute(job, config): + global jobconfig + jobconfig = config + + print + print "============================================================================" + print " EXECUTING: %s" % job + print "============================================================================" + + generateScript() + + + + + + +###################################################################### +# CORE: GENERATORS +###################################################################### + +def getJobConfig(key, default=None): + global jobconfig + + if jobconfig.has_key(key): + return jobconfig[key] + else: + return default + + +def generateScript(): + global classes + global modules + global verbose + global quiet + + + # + # INITIALIZATION PHASE + # + + # Class paths + classPaths = getJobConfig("classPath") + + # Script names + buildScript = getJobConfig("buildScript") + sourceScript = getJobConfig("sourceScript") + apiPath = getJobConfig("apiPath") + + # Dynamic dependencies + dynLoadDeps = getJobConfig("require", {}) + dynRunDeps = getJobConfig("use", {}) + + # Variants data + # TODO: Variants for source -> Not possible + userVariants = getJobConfig("variants", {}) + + # Part support (has priority) + userParts = getJobConfig("parts", {}) + + # Build relevant post processing + buildProcess = getJobConfig("buildProcess", []) + + userInclude = getJobConfig("include", []) + userExclude = getJobConfig("exclude", []) + + collapseParts = getJobConfig("collapseParts", []) + optimizeLatency = getJobConfig("optimizeLatency") + + + + if len(userParts) > 0: + execMode = "parts" + else: + execMode = "normal" + + + + # + # SCAN PHASE + # + + # Scan for classes and modules + scanClassPaths(classPaths) + scanModules() + + + + + # + # PREPROCESS PHASE: INCLUDE/EXCLUDE + # + + # Auto include all when nothing defined + if execMode == "normal" and len(userInclude) == 0: + print " - Automatically including all available classes" + userInclude.append("*") + + + + print ">>> Preparing include/exclude configuration..." + smartInclude, explicitInclude = _splitIncludeExcludeList(userInclude) + smartExclude, explicitExclude = _splitIncludeExcludeList(userExclude) + + + # Configuration feedback + if not quiet: + print " - Including %s items smart, %s items explicit" % (len(smartInclude), len(explicitInclude)) + print " - Excluding %s items smart, %s items explicit" % (len(smartExclude), len(explicitExclude)) + + if len(userExclude) > 0: + print " - Warning: Excludes may break code!" + + if len(explicitInclude) > 0: + print " - Warning: Explicit included classes may not work" + + + + + + + # Resolve modules/regexps + print " - Resolving modules/regexps..." + smartInclude = resolveComplexDefs(smartInclude) + explicitInclude = resolveComplexDefs(explicitInclude) + smartExclude = resolveComplexDefs(smartExclude) + explicitExclude = resolveComplexDefs(explicitExclude) + + + + + + # + # PREPROCESS PHASE: VIEWS + # + + if execMode == "parts": + print + print ">>> Preparing part configuration..." + + # Build bitmask ids for parts + if verbose: + print + + print " - Assigning bits to parts..." + + # References partId -> bitId of that part + partBits = {} + + partPos = 0 + for partId in userParts: + partBit = 1<<partPos + + if verbose: + print " - Part #%s => %s" % (partId, partBit) + + partBits[partId] = partBit + partPos += 1 + + # Resolving modules/regexps + print " - Resolving part modules/regexps..." + partClasses = {} + for partId in userParts: + partClasses[partId] = resolveComplexDefs(userParts[partId]) + + + + + # + # EXECUTION PHASE + # + + sets = _computeVariantCombinations(userVariants) + for pos, variants in enumerate(sets): + print + print "----------------------------------------------------------------------------" + print " PROCESSING VARIANT SET %s/%s" % (pos+1, len(sets)) + print "----------------------------------------------------------------------------" + if not quiet and len(variants) > 0: + for entry in variants: + print " - %s = %s" % (entry["id"], entry["value"]) + print "----------------------------------------------------------------------------" + + + # Detect dependencies + print ">>> Resolving application dependencies..." + includeDict = resolveDependencies(smartInclude, smartExclude, dynLoadDeps, dynRunDeps, variants) + + + # Explicit include/exclude + if len(explicitInclude) > 0 or len(explicitExclude) > 0: + print ">>> Processing explicitely configured includes/excludes..." + for entry in explicitInclude: + includeDict[entry] = True + + for entry in explicitExclude: + if includeDict.has_key(entry): + del includeDict[entry] + + + # Detect optionals + if verbose: + optionals = getOptionals(includeDict) + if len(optionals) > 0: + print ">>> These optional classes may be useful:" + for entry in optionals: + print " - %s" % entry + + + if apiPath != None: + storeApi(includeDict, dynLoadDeps, dynRunDeps, apiPath) + + + if buildScript != None: + if execMode == "parts": + processParts(partClasses, partBits, includeDict, dynLoadDeps, dynRunDeps, variants, collapseParts, optimizeLatency, buildScript, buildProcess) + else: + sys.stdout.write(">>> Compiling classes:") + sys.stdout.flush() + packageFileName = buildScript + "_$variants_$process.js" + packageSize = storeCompiledPackage(includeDict, packageFileName, dynLoadDeps, dynRunDeps, variants, buildProcess) + print " - Done: %s" % packageSize + + + + + + + +###################################################################### +# CORE: CACHE SUPPORT +###################################################################### + +# Improved version of the one in filetool module + +def readCache(id, segment, dep): + global jobconfig + + cachePath = jobconfig["cachePath"] + + if not cachePath.endswith(os.sep): + cachePath += os.sep + + filetool.directory(cachePath) + + fileModTime = os.stat(dep).st_mtime + + try: + cacheModTime = os.stat(cachePath + id + "-" + segment).st_mtime + except OSError: + cacheModTime = 0 + + # Out of date check + if fileModTime > cacheModTime: + return None + + try: + return cPickle.load(open(cachePath + id + "-" + segment, 'rb')) + + except (IOError, EOFError, cPickle.PickleError, cPickle.UnpicklingError): + print ">>> Could not read cache from %s" % cachePath + return None + + + +def writeCache(id, segment, content): + global jobconfig + + cachePath = jobconfig["cachePath"] + + if not cachePath.endswith(os.sep): + cachePath += os.sep + + filetool.directory(cachePath) + + try: + cPickle.dump(content, open(cachePath + id + "-" + segment, 'wb'), 2) + + except (IOError, EOFError, cPickle.PickleError, cPickle.PicklingError): + print ">>> Could not store cache to %s" % cachePath + sys.exit(1) + + + + + + +###################################################################### +# CORE: SESSION STABLE HASH CODE SUPPORT +###################################################################### + +# calculates a hash code (simple incrementer) +# cache all already calculates inputs for the next session using pickle +# to keep hash codes identical between different sessions +def toHashCode(id): + global classes + global hashes + + try: + hashes = cPickle.load(open(cachePath + "hashes", 'rb')) + except (IOError, EOFError, cPickle.PickleError, cPickle.UnpicklingError): + hashes = {} + + if not hashes.has_key(id): + hashes[id] = mapper.convert(len(hashes)) + + try: + cPickle.dump(hashes, open(cachePath + "hashes", 'wb'), 2) + except (IOError, EOFError, cPickle.PickleError, cPickle.UnpicklingError): + print ">>> Could not store hash cache: %s" % cachePath + sys.exit(1) + + return hashes[id] + + + + + +###################################################################### +# CORE: PROGRESS DISPLAY +###################################################################### + +def printProgress(pos, length): + global quiet + + if quiet: + return + + # starts normally at null, but this is not useful here + # also the length is normally +1 the real size + pos += 1 + + thisstep = 10 * pos / length + prevstep = 10 * (pos-1) / length + + if thisstep != prevstep: + sys.stdout.write(" %s%%" % (thisstep * 10)) + sys.stdout.flush() + + if pos == length: + sys.stdout.write("\n") + sys.stdout.flush() + + + + + +###################################################################### +# API: DATA SUPPORT +###################################################################### + +def getApi(id): + global classes + + entry = classes[id] + path = entry["path"] + + cache = readCache(id, "api", path) + if cache != None: + return cache + + tree = getTree(id) + + if verbose: + print " - Generating API data: %s..." % id + + (apidata, hasError) = api.createDoc(tree) + + writeCache(id, "api", apidata) + + return apidata + + + +def storeApi(includeDict, dynLoadDeps, dynRunDeps, apiPath): + docTree = tree.Node("doctree") + todo = includeDict.keys() + length = len(todo) + + sys.stdout.write(">>> Generating API data:") + sys.stdout.flush() + for pos, id in enumerate(todo): + printProgress(pos, length) + _mergeApiNodes(docTree, getApi(id)) + + print " - Postprocessing..." + api.postWorkPackage(docTree, docTree) + + print " - Storing..." + packages = api.packagesToJsonString(docTree, "", " ", "\n") + filetool.save(os.path.join(apiPath, "apidata.js"), packages) + + for classData in api.classNodeIterator(docTree): + classContent = tree.nodeToJsonString(classData, "", " ", "\n") + fileName = os.path.join(apiPath, classData.get("fullName") + ".js") + filetool.save(fileName, classContent) + + print ">>> Done" + + + +def _mergeApiNodes(target, source): + if source.hasAttributes(): + attr = source.attributes + for key in attr: + # Special handling for attributes which stores a list (this is BTW quite ugly) + if key in ["childClasses", "includer", "mixins", "implementations"] and target.get(key, False) != None: + target.set(key, "%s,%s" % (target.get(key), attr[key])) + else: + target.set(key, attr[key]) + + if source.hasChildren(): + # copy to keep length while iterating + children = source.children[:] + + for child in children: + # print "Child: %s" % child.type + + # no such type => append + if not target.hasChild(child.type): + # print "=> direct append" + target.addChild(child) + + else: + # looking for identical child (e.g. equal name etc.) + identical = _findIdenticalChild(target, child) + if identical: + # print "=> identical merge" + _mergeApiNodes(identical, child) + + else: + # print "=> fallback append" + target.addChild(child) + + + +def _findIdenticalChild(node, search): + if node.hasChildren(): + for child in node.children: + if _isNodeIdentical(child, search): + return child + + return None + + + +def _isNodeIdentical(nodeA, nodeB): + if nodeA.type == nodeB.type: + if not nodeA.hasAttributes() and not nodeB.hasAttributes(): + return True + + if nodeA.type in [ "method", "param", "property", "event" ]: + return nodeA.get("name") == nodeB.get("name") + + if nodeA.type in [ "class", "package", "interface", "mixin" ]: + return nodeA.get("fullName") == nodeB.get("fullName") + + return False + + + + + + + +###################################################################### +# META DATA SUPPORT +###################################################################### + +def getMeta(id): + global classes + + entry = classes[id] + path = entry["path"] + + cache = readCache(id, "meta", path) + if cache != None: + return cache + + meta = {} + category = entry["category"] + + if category == "qx.doc": + pass + + elif category == "qx.locale": + meta["loadtimeDeps"] = ["qx.locale.Locale", "qx.locale.Manager"] + + elif category == "qx.impl": + content = filetool.read(path, entry["encoding"]) + + meta["loadtimeDeps"] = _extractQxLoadtimeDeps(content, id) + meta["runtimeDeps"] = _extractQxRuntimeDeps(content, id) + meta["optionalDeps"] = _extractQxOptionalDeps(content) + meta["ignoreDeps"] = _extractQxIgnoreDeps(content) + + meta["modules"] = _extractQxModules(content) + meta["resources"] = _extractQxResources(content) + meta["embeds"] = _extractQxEmbeds(content) + + writeCache(id, "meta", meta) + + return meta + + + +def _extractQxLoadtimeDeps(data, fileId=""): + deps = [] + + for item in config.QXHEAD["require"].findall(data): + if item == fileId: + print " - Error: Self-referring load dependency: %s" % item + sys.exit(1) + else: + deps.append(item) + + return deps + + + +def _extractQxRuntimeDeps(data, fileId=""): + deps = [] + + for item in config.QXHEAD["use"].findall(data): + if item == fileId: + print " - Self-referring runtime dependency: %s" % item + else: + deps.append(item) + + return deps + + + +def _extractQxOptionalDeps(data): + deps = [] + + # Adding explicit requirements + for item in config.QXHEAD["optional"].findall(data): + if not item in deps: + deps.append(item) + + return deps + + + +def _extractQxIgnoreDeps(data): + ignores = [] + + # Adding explicit requirements + for item in config.QXHEAD["ignore"].findall(data): + if not item in ignores: + ignores.append(item) + + return ignores + + + +def _extractQxModules(data): + mods = [] + + for item in config.QXHEAD["module"].findall(data): + if not item in mods: + mods.append(item) + + return mods + + + +def _extractQxResources(data): + res = [] + + for item in config.QXHEAD["resource"].findall(data): + res.append({ "namespace" : item[0], "id" : item[1], "entry" : item[2] }) + + return res + + + +def _extractQxEmbeds(data): + emb = [] + + for item in config.QXHEAD["embed"].findall(data): + emb.append({ "namespace" : item[0], "id" : item[1], "entry" : item[2] }) + + return emb + + + + + + + +###################################################################### +# TOKEN/TREE SUPPORT +###################################################################### + +def getTokens(id): + global classes + global verbose + + cache = readCache(id, "tokens", classes[id]["path"]) + if cache != None: + return cache + + if verbose: + print " - Generating tokens: %s..." % id + tokens = tokenizer.parseFile(classes[id]["path"], id, classes[id]["encoding"]) + + writeCache(id, "tokens", tokens) + return tokens + + + +def getLength(id): + return len(getTokens(id)) + + + +def getTree(id): + global classes + global verbose + + cache = readCache(id, "tree", classes[id]["path"]) + if cache != None: + return cache + + tokens = getTokens(id) + + if verbose: + print " - Generating tree: %s..." % id + tree = treegenerator.createSyntaxTree(tokens) + + writeCache(id, "tree", tree) + return tree + + + +def getVariantsTree(id, variants): + global classes + global verbose + + variantsId = generateVariantCombinationId(variants) + + if variantsId != "": + variantsId = "-" + variantsId + + cache = readCache(id, "tree" + variantsId, classes[id]["path"]) + if cache != None: + return cache + + # Generate map + variantsMap = {} + for entry in variants: + variantsMap[entry["id"]] = entry["value"] + + # Copy tree to work with + tree = copy.deepcopy(getTree(id)) + + if verbose: + print " - Select variants: %s..." % id + + # Call variant optimizer + variantoptimizer.search(tree, variantsMap, id) + + # Store result into cache + writeCache(id, "tree" + variantsId, tree) + + return tree + + + + + + + + + +###################################################################### +# COMMON COMPILED PKG SUPPORT +###################################################################### + +def storeCompiledPackage(includeDict, packageFileName, loadDeps, runDeps, variants, buildProcess): + # Compiling classes + sortedClasses = sortClasses(includeDict, loadDeps, runDeps, variants) + compiledContent = compileClasses(sortedClasses, variants, buildProcess) + + # Pre storage calculations + variantsId = generateVariantCombinationId(variants) + processId = generateProcessCombinationId(buildProcess) + + packageFileName = packageFileName.replace("$variants", variantsId) + packageFileName = packageFileName.replace("$process", processId) + + # Saving compiled content + filetool.save(packageFileName, compiledContent) + return getContentSize(compiledContent) + + + +def getContentSize(content): + # Convert to utf-8 first + uni = unicode(content).encode("utf-8") + + # Calculate sizes + origSize = len(uni) / 1024 + compressedSize = len(zlib.compress(uni, 9)) / 1024 + + return "%sKB / %sKB" % (origSize, compressedSize) + + + +def _splitIncludeExcludeList(input): + intelli = [] + explicit = [] + + for entry in input: + if entry[0] == "=": + explicit.append(entry[1:]) + else: + intelli.append(entry) + + return intelli, explicit + + + +def _findCombinations(a): + result = [[]] + + for x in a: + t = [] + for y in x: + for i in result: + t.append(i+[y]) + result = t + + return result + + + +def _computeVariantCombinations(variants): + variantPossibilities = [] + for variantId in variants: + innerList = [] + for variantValue in variants[variantId]: + innerList.append({"id" : variantId, "value" : variantValue}) + variantPossibilities.append(innerList) + + return _findCombinations(variantPossibilities) + + + +def generateVariantCombinationId(selected): + def _compare(x, y): + if x["id"] > y["id"]: + return 1 + + if x["id"] < y["id"]: + return -1 + + return 0 + + sortedList = [] + sortedList.extend(selected) + sortedList.sort(_compare) + + sortedString = [] + for entry in sortedList: + sortedString.append("(" + entry["id"].replace(".", "") + "_" + entry["value"] + ")") + + return "_".join(sortedString) + + + + + + + +###################################################################### +# VIEW/PACKAGE SUPPORT +###################################################################### + +def processParts(partClasses, partBits, includeDict, loadDeps, runDeps, variants, collapseParts, optimizeLatency, buildScript, buildProcess): + global classes + global verbose + global quiet + + + # Caching dependencies of each part + if not quiet: + print + + print ">>> Resolving part dependencies..." + partDeps = {} + length = len(partClasses.keys()) + for pos, partId in enumerate(partClasses.keys()): + if not quiet: + print " - Part #%s..." % partId + + # Exclude all features of other parts + # and handle dependencies the smart way => + # also exclude classes only needed by the + # already excluded features + partExcludes = [] + for subPartId in partClasses: + if subPartId != partId: + partExcludes.extend(partClasses[subPartId]) + + # Finally resolve the dependencies + localDeps = resolveDependencies(partClasses[partId], partExcludes, loadDeps, runDeps, variants) + + + # Remove all dependencies which are not included in the full list + if len(includeDict) > 0: + depKeys = localDeps.keys() + for dep in depKeys: + if not dep in includeDict: + del localDeps[dep] + + if not quiet: + print " - Needs %s classes" % len(localDeps) + + partDeps[partId] = localDeps + + + + # Assign classes to packages + if not quiet: + print + + print ">>> Assigning classes to packages..." + + # References packageId -> class list + packageClasses = {} + + # References packageId -> bit number e.g. 4=1, 5=2, 6=2, 7=3 + packageBitCounts = {} + + for classId in classes: + packageId = 0 + bitCount = 0 + + # Iterate through the parts use needs this class + for partId in partClasses: + if classId in partDeps[partId]: + packageId += partBits[partId] + bitCount += 1 + + # Ignore unused classes + if packageId == 0: + continue + + # Create missing data structure + if not packageClasses.has_key(packageId): + packageClasses[packageId] = [] + packageBitCounts[packageId] = bitCount + + # Finally store the class to the package + packageClasses[packageId].append(classId) + + + + + # Assign packages to parts + print ">>> Assigning packages to parts..." + partPackages = {} + + for partId in partClasses: + partBit = partBits[partId] + + for packageId in packageClasses: + if packageId&partBit: + if not partPackages.has_key(partId): + partPackages[partId] = [] + + partPackages[partId].append(packageId) + + # Be sure that the part package list is in order to the package priorit + _sortPackageIdsByPriority(partPackages[partId], packageBitCounts) + + + + + # User feedback + _printPartStats(packageClasses, partPackages) + + + + # Support for package collapsing + # Could improve latency when initial loading an application + # Merge all packages of a specific part into one (also supports multiple parts) + # Hint: Part packages are sorted by priority, this way we can + # easily merge all following packages with the first one, because + # the first one is always the one with the highest priority + if len(collapseParts) > 0: + if not quiet: + print + + collapsePos = 0 + print ">>> Collapsing parts..." + for partId in collapseParts: + if not quiet: + print " - Collapsing part #%s(%s)..." % (partId, collapsePos) + + collapsePackage = partPackages[partId][collapsePos] + for packageId in partPackages[partId][collapsePos+1:]: + if not quiet: + print " - Merge package #%s into #%s" % (packageId, collapsePackage) + + _mergePackage(packageId, collapsePackage, partClasses, partPackages, packageClasses) + + collapsePos += 1 + + # User feedback + _printPartStats(packageClasses, partPackages) + + + # Support for merging small packages + # Hint1: Based on the token length which is a bit strange but a good + # possibility to get the not really correct filesize in an ultrafast way + # More complex code and classes generally also have more tokens in them + # Hint2: The first common package before the selected package between two + # or more parts is allowed to merge with. As the package which should be merged + # may have requirements these must be solved. The easiest way to be sure regarding + # this issue, is to look out for another common package. The package for which we + # are looking must have requirements in all parts so these must be solved by all parts + # so there must be another common package. Hardly to describe... hope this makes some sense + if optimizeLatency != None and optimizeLatency != 0: + smallPackages = [] + + # Start at the end with the priority sorted list + sortedPackageIds = _sortPackageIdsByPriority(_dictToHumanSortedList(packageClasses), packageBitCounts) + sortedPackageIds.reverse() + + if not quiet: + print + print ">>> Analysing package sizes..." + if not quiet: + print " - Optimize at %s tokens" % optimizeLatency + + for packageId in sortedPackageIds: + packageLength = 0 + + for classId in packageClasses[packageId]: + packageLength += getLength(classId) + + if packageLength >= optimizeLatency: + if not quiet: + print " - Package #%s has %s tokens" % (packageId, packageLength) + continue + else: + if not quiet: + print " - Package #%s has %s tokens => trying to optimize" % (packageId, packageLength) + + collapsePackage = _getPreviousCommonPackage(packageId, partPackages, packageBitCounts) + if collapsePackage != None: + if not quiet: + print " - Merge package #%s into #%s" % (packageId, collapsePackage) + _mergePackage(packageId, collapsePackage, partClasses, partPackages, packageClasses) + + # User feedback + _printPartStats(packageClasses, partPackages) + + + + # Compile files... + packageLoaderContent = "" + sortedPackageIds = _sortPackageIdsByPriority(_dictToHumanSortedList(packageClasses), packageBitCounts) + variantsId = generateVariantCombinationId(variants) + processId = generateProcessCombinationId(buildProcess) + + if not quiet: + print + + + print ">>> Creating packages..." + for packageId in sortedPackageIds: + sys.stdout.write(" - Compiling package #%s:" % packageId) + sys.stdout.flush() + + packageFileName = buildScript + "_$variants_$process_%s.js" % packageId + packageSize = storeCompiledPackage(packageClasses[packageId], packageFileName, loadDeps, runDeps, variants, buildProcess) + print " - Done: %s" % packageSize + + # TODO: Make prefix configurable + prefix = "script/" + packageLoaderContent += "document.write('<script type=\"text/javascript\" src=\"%s\"></script>');\n" % (prefix + packageFileName) + + + print ">>> Storing package loader script..." + packageLoader = "%s_%s_%s.js" % (buildScript, variantsId, processId) + filetool.save(packageLoader, packageLoaderContent) + + + +def _sortPackageIdsByPriority(packageIds, packageBitCounts): + def _cmpPackageIds(pkgId1, pkgId2): + if packageBitCounts[pkgId2] > packageBitCounts[pkgId1]: + return 1 + elif packageBitCounts[pkgId2] < packageBitCounts[pkgId1]: + return -1 + + return pkgId2 - pkgId1 + + packageIds.sort(_cmpPackageIds) + + return packageIds + + + +def _getPreviousCommonPackage(searchId, partPackages, packageBitCounts): + relevantParts = [] + relevantPackages = [] + + for partId in partPackages: + packages = partPackages[partId] + if searchId in packages: + relevantParts.append(partId) + relevantPackages.extend(packages[:packages.index(searchId)]) + + # Sorted by priority, but start from end + _sortPackageIdsByPriority(relevantPackages, packageBitCounts) + relevantPackages.reverse() + + # Check if a package is available identical times to the number of parts + for packageId in relevantPackages: + if relevantPackages.count(packageId) == len(relevantParts): + return packageId + + return None + + + +def _printPartStats(packageClasses, partPackages): + global verbose + + if not verbose: + return + + packageIds = _dictToHumanSortedList(packageClasses) + + print + print ">>> Content of packages(%s):" % len(packageIds) + for packageId in packageIds: + print " - Package #%s contains %s classes" % (packageId, len(packageClasses[packageId])) + + print + print ">>> Content of parts(%s):" % len(partPackages) + for partId in partPackages: + print " - Part #%s uses these packages: %s" % (partId, _intListToString(partPackages[partId])) + + + +def _dictToHumanSortedList(input): + output = [] + for key in input: + output.append(key) + output.sort() + output.reverse() + + return output + + + +def _mergePackage(replacePackage, collapsePackage, partClasses, partPackages, packageClasses): + # Replace other package content + for partId in partClasses: + partContent = partPackages[partId] + + if replacePackage in partContent: + # Store collapse package at the place of the old value + partContent[partContent.index(replacePackage)] = collapsePackage + + # Remove duplicate (may be, but only one) + if partContent.count(collapsePackage) > 1: + partContent.reverse() + partContent.remove(collapsePackage) + partContent.reverse() + + # Merging collapsed packages + packageClasses[collapsePackage].extend(packageClasses[replacePackage]) + del packageClasses[replacePackage] + + + +def _intListToString(input): + result = "" + for entry in input: + result += "#%s, " % entry + + return result[:-2] + + + + + + +###################################################################### +# MODULE/REGEXP SUPPORT +###################################################################### + +def resolveComplexDefs(entries): + global modules + global classes + + content = [] + + for entry in entries: + if entry in modules: + content.extend(modules[entry]) + else: + regexp = textutil.toRegExp(entry) + + for className in classes: + if regexp.search(className): + if not className in content: + # print "Resolved: %s with %s" % (entry, className) + content.append(className) + + return content + + + + + + +###################################################################### +# SUPPORT FOR OPTIONAL CLASSES/FEATURES +###################################################################### + +def getOptionals(classes): + opt = {} + + for id in classes: + for sub in getMeta(id)["optionalDeps"]: + if not sub in classes: + opt[sub] = True + + return opt + + + + + + +###################################################################### +# COMPILER SUPPORT +###################################################################### + +def compileClasses(todo, variants, process): + content = "" + length = len(todo) + + for pos, id in enumerate(todo): + printProgress(pos, length) + content += getCompiled(id, variants, process) + + return content + + +def _compileClassHelper(restree): + # Emulate options + parser = optparse.OptionParser() + parser.add_option("--p1", action="store_true", dest="prettyPrint", default=False) + parser.add_option("--p2", action="store_true", dest="prettypIndentString", default=" ") + parser.add_option("--p3", action="store_true", dest="prettypCommentsInlinePadding", default=" ") + parser.add_option("--p4", action="store_true", dest="prettypCommentsTrailingCommentCols", default="") + + (options, args) = parser.parse_args([]) + + return compiler.compile(restree, options) + + +def getCompiled(id, variants, process): + global classes + global verbose + + variantsId = generateVariantCombinationId(variants) + processId = generateProcessCombinationId(process) + + if variantsId != "": + variantsId = "-" + variantsId + + if processId != "": + processId = "-" + processId + + cache = readCache(id, "compiled" + variantsId + processId, classes[id]["path"]) + if cache != None: + return cache + + tokens = getTokens(id) + + tree = copy.deepcopy(getVariantsTree(id, variants)) + + if verbose: + print " - Postprocessing tree: %s..." % id + + tree = _postProcessHelper(tree, id, process, variants) + + if verbose: + print " - Compiling tree: %s..." % id + + compiled = _compileClassHelper(tree) + + writeCache(id, "compiled" + variantsId + processId, compiled) + return compiled + + +def _postProcessHelper(tree, id, process, variants): + global verbose + global quiet + + if "optimize-basecalls" in process: + if verbose: + print " - Optimize base calls..." + baseCallOptimizeHelper(tree, id, variants) + + if "optimize-variables" in process: + if verbose: + print " - Optimize local variables..." + variableOptimizeHelper(tree, id, variants) + + if "optimize-privates" in process: + if verbose: + print " - Optimize privates..." + privateOptimizeHelper(tree, id, variants) + + if "optimize-strings" in process: + if verbose: + print " - Optimize strings..." + stringOptimizeHelper(tree, id, variants) + + return tree + + +def generateProcessCombinationId(process): + process = copy.copy(process) + process.sort() + + return "[%s]" % ("-".join(process)) + + +def baseCallOptimizeHelper(tree, id, variants): + basecalloptimizer.patch(tree) + + +def variableOptimizeHelper(tree, id, variants): + variableoptimizer.search(tree, [], 0, 0, "$") + + +def privateOptimizeHelper(tree, id, variants): + unique = toHashCode(id) + privateoptimizer.patch(unique, tree, {}) + + +def stringOptimizeHelper(tree, id, variants): + global verbose + global quiet + + # Do not optimize strings for non-mshtml clients + clientValue = getVariantValue(variants, "qx.client") + if clientValue != None and clientValue != "mshtml": + return + + # TODO: Customize option for __SS__ + + stringMap = stringoptimizer.search(tree) + stringList = stringoptimizer.sort(stringMap) + + stringoptimizer.replace(tree, stringList, "__SS__") + + # Build JS string fragments + stringStart = "(function(){" + stringReplacement = "var " + stringoptimizer.replacement(stringList, "__SS__") + stringStop = "})();" + + # Compile wrapper node + wrapperNode = treeutil.compileString(stringStart+stringReplacement+stringStop) + + # Reorganize structure + funcBody = wrapperNode.getChild("operand").getChild("group").getChild("function").getChild("body").getChild("block") + if tree.hasChildren(): + for child in copy.copy(tree.children): + tree.removeChild(child) + funcBody.addChild(child) + + # Add wrapper to tree + tree.addChild(wrapperNode) + + +def getVariantValue(variants, key): + for entry in variants: + if entry["id"] == key: + return entry["value"] + + return None + + + + + + +###################################################################### +# CLASS DEPENDENCY SUPPORT +###################################################################### + +def resolveDependencies(add, block, loadDeps, runDeps, variants): + result = {} + + for entry in add: + _resolveDependenciesRecurser(entry, result, block, loadDeps, runDeps, variants) + + return result + + +def _resolveDependenciesRecurser(add, result, block, loadDeps, runDeps, variants): + global classes + + # check if already in + if result.has_key(add): + return + + # add self + + result[add] = True + + # reading dependencies + deps = getCombinedDeps(add, loadDeps, runDeps, variants) + + # process lists + for sub in deps["load"]: + if not result.has_key(sub) and not sub in block: + _resolveDependenciesRecurser(sub, result, block, loadDeps, runDeps, variants) + + for sub in deps["run"]: + if not result.has_key(sub) and not sub in block: + _resolveDependenciesRecurser(sub, result, block, loadDeps, runDeps, variants) + + +def getCombinedDeps(id, loadDeps, runDeps, variants): + # init lists + loadFinal = [] + runFinal = [] + + # add static dependencies + static = getDeps(id, variants) + loadFinal.extend(static["load"]) + runFinal.extend(static["run"]) + + # add dynamic dependencies + if loadDeps.has_key(id): + loadFinal.extend(loadDeps[id]) + + if runDeps.has_key(id): + runFinal.extend(runDeps[id]) + + # return dict + return { + "load" : loadFinal, + "run" : runFinal + } + + +def getDeps(id, variants): + global classes + global verbose + + variantsId = generateVariantCombinationId(variants) + + cache = readCache(id, "deps" + variantsId, classes[id]["path"]) + if cache != None: + return cache + + # Notes: + # load time = before class = require + # runtime = after class = use + + if verbose: + print " - Gathering dependencies: %s" % id + + load = [] + run = [] + + # Read meta data + + meta = getMeta(id) + metaLoad = _readDictKey(meta, "loadtimeDeps", []) + metaRun = _readDictKey(meta, "runtimeDeps", []) + metaOptional = _readDictKey(meta, "optionalDeps", []) + metaIgnore = _readDictKey(meta, "ignoreDeps", []) + + # Process meta data + load.extend(metaLoad) + run.extend(metaRun) + + # Read content data + (autoLoad, autoRun) = _analyzeClassDeps(id, variants) + + # Process content data + if not "auto-require" in metaIgnore: + for entry in autoLoad: + if entry in metaOptional: + pass + elif entry in load: + if verbose: + print " - #require(%s) is auto-detected" % entry + else: + load.append(entry) + + if not "auto-use" in metaIgnore: + for entry in autoRun: + if entry in metaOptional: + pass + elif entry in load: + pass + elif entry in run: + if verbose: + print " - #use(%s) is auto-detected" % entry + else: + run.append(entry) + + # Build data structure + deps = { + "load" : load, + "run" : run + } + + writeCache(id, "deps" + variantsId, deps) + + return deps + + +def _readDictKey(data, key, default=None): + if data.has_key(key): + return data[key] + + return default + + +def _analyzeClassDeps(id, variants): + global classes + + tree = getVariantsTree(id, variants) + loadtime = [] + runtime = [] + + _analyzeClassDepsNode(id, tree, loadtime, runtime, False) + + return loadtime, runtime + + +def _analyzeClassDepsNode(id, node, loadtime, runtime, inFunction): + global classes + + if node.type == "variable": + if node.hasChildren: + assembled = "" + first = True + + for child in node.children: + if child.type == "identifier": + if not first: + assembled += "." + + assembled += child.get("name") + first = False + + if assembled != id and classes.has_key(assembled): + if inFunction: + target = runtime + else: + target = loadtime + + if assembled in target: + return + + target.append(assembled) + + else: + assembled = "" + break + + # treat dependencies in defer as requires + if assembled == "qx.Class.define": + if node.parent.type == "operand" and node.parent.parent.type == "call": + deferNode = treeutil.selectNode(node, "../../params/2/keyvalue[@key='defer']/value/function/body/block") + if deferNode != None: + _analyzeClassDepsNode(id, deferNode, loadtime, runtime, False) + + elif node.type == "body" and node.parent.type == "function": + inFunction = True + + if node.hasChildren(): + for child in node.children: + _analyzeClassDepsNode(id, child, loadtime, runtime, inFunction) + + + + + + + + +###################################################################### +# CLASS SORT SUPPORT +###################################################################### + +def sortClasses(input, loadDeps, runDeps, variants): + sorted = [] + + for entry in input: + _sortClassesRecurser(entry, input, sorted, loadDeps, runDeps, variants) + + return sorted + + +def _sortClassesRecurser(id, available, sorted, loadDeps, runDeps, variants): + global classes + + if id in sorted: + return + + # reading dependencies + deps = getCombinedDeps(id, loadDeps, runDeps, variants) + + # process loadtime requirements + for entry in deps["load"]: + if entry in available and not entry in sorted: + _sortClassesRecurser(entry, available, sorted, loadDeps, runDeps, variants) + + if id in sorted: + return + + # print " - Adding: %s" % id + sorted.append(id) + + # process runtime requirements + for entry in deps["run"]: + if entry in available and not entry in sorted: + _sortClassesRecurser(entry, available, sorted, loadDeps, runDeps, variants) + + + + + +###################################################################### +# CLASS PATH SUPPORT +###################################################################### + +def scanModules(): + global classes + global modules + global quiet + + modules = {} + + print ">>> Searching for module definitions..." + for id in classes: + if classes[id]["category"] == "qx.impl": + for mod in getMeta(id)["modules"]: + if not modules.has_key(mod): + modules[mod] = [] + + modules[mod].append(id) + + if not quiet: + print " - Found %s modules" % len(modules) + print + + +def scanClassPaths(paths): + global classes + global quiet + + classes = {} + + print ">>> Scanning class paths..." + for path in paths: + _addClassPath(path) + + if not quiet: + print + + return classes + + +def _addClassPath(classPath, encoding="utf-8"): + global classes + global quiet + + if not quiet: + print " - Scanning: %s" % classPath + + implCounter = 0 + docCounter = 0 + localeCounter = 0 + + for root, dirs, files in os.walk(classPath): + + # Filter ignored directories + for ignoredDir in config.DIRIGNORE: + if ignoredDir in dirs: + dirs.remove(ignoredDir) + + # Searching for files + for fileName in files: + if os.path.splitext(fileName)[1] == config.JSEXT and not fileName.startswith("."): + filePath = os.path.join(root, fileName) + filePathId = filePath.replace(classPath + os.sep, "").replace(config.JSEXT, "").replace(os.sep, ".") + fileContent = filetool.read(filePath, encoding) + fileCategory = "unknown" + + if fileName == "__init__.js": + fileContentId = filePathId + fileCategory = "qx.doc" + docCounter += 1 + + else: + fileContentId = _extractQxClassContentId(fileContent) + + if fileContentId == None: + fileContentId = _extractQxLocaleContentId(fileContent) + + if fileContentId != None: + fileCategory = "qx.locale" + localeCounter += 1 + + else: + fileCategory = "qx.impl" + implCounter += 1 + + if filePathId != fileContentId: + print " - Mismatching IDs in file: %s" % filePath + print " Detail: %s != %s" % (filePathId, fileContentId) + + if fileCategory == "unknown": + print " - Invalid file: %s" % filePath + sys.exit(1) + + fileId = filePathId + + classes[fileId] = { + "path" : filePath, + "encoding" : encoding, + "classPath" : classPath, + "category" : fileCategory, + "id" : fileId, + "contentId" : fileContentId, + "pathId" : filePathId + } + + if not quiet: + print " - Found: %s impl, %s doc, %s locale" % (implCounter, docCounter, localeCounter) + + +def _extractQxClassContentId(data): + classDefine = re.compile('qx.(Bootstrap|List|Class|Mixin|Interface|Theme).define\s*\(\s*["\']([\.a-zA-Z0-9_-]+)["\']?', re.M) + + for item in classDefine.findall(data): + return item[1] + + return None + + +def _extractQxLocaleContentId(data): + # 0.8 style + localeDefine = re.compile('qx.Locale.define\s*\(\s*["\']([\.a-zA-Z0-9_-]+)["\']?', re.M) + + for item in localeDefine.findall(data): + return item + + # 0.7.x compat + localeDefine = re.compile('qx.locale.Locale.define\s*\(\s*["\']([\.a-zA-Z0-9_-]+)["\']?', re.M) + + for item in localeDefine.findall(data): + return item + + return None + + + + + + + +###################################################################### +# MAIN LOOP +###################################################################### + +if __name__ == '__main__': + try: + main() + + except KeyboardInterrupt: + print + print " * Keyboard Interrupt" + sys.exit(1) + This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-24 09:11:36
|
Revision: 10655 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10655&view=rev Author: thron7 Date: 2007-10-24 02:11:27 -0700 (Wed, 24 Oct 2007) Log Message: ----------- Make generator2.py executable. Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-24 07:49:01 UTC (rev 10654) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-24 09:11:27 UTC (rev 10655) @@ -83,7 +83,7 @@ import sys, re, os, optparse, math, cPickle, copy, sets, zlib # reconfigure path to import own modules from modules subfolder -sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])), "modules")) +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])), "modules", "generator2")) import config, tokenizer, tree, treegenerator, treeutil, optparseext, filetool import compiler, textutil, mapper This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-24 09:18:40
|
Revision: 10656 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10656&view=rev Author: thron7 Date: 2007-10-24 02:18:37 -0700 (Wed, 24 Oct 2007) Log Message: ----------- Fixed sys.path. Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-24 09:11:27 UTC (rev 10655) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-24 09:18:37 UTC (rev 10656) @@ -83,7 +83,9 @@ import sys, re, os, optparse, math, cPickle, copy, sets, zlib # reconfigure path to import own modules from modules subfolder -sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])), "modules", "generator2")) +script_path = os.path.dirname(os.path.abspath(sys.argv[0])) +sys.path.insert(0, os.path.join(script_path, "modules")) +sys.path.insert(0, os.path.join(script_path, "generator2")) import config, tokenizer, tree, treegenerator, treeutil, optparseext, filetool import compiler, textutil, mapper This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-24 09:58:09
|
Revision: 10658 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10658&view=rev Author: thron7 Date: 2007-10-24 02:58:06 -0700 (Wed, 24 Oct 2007) Log Message: ----------- Fixed missing cachePath in toHashCode(). Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-24 09:48:57 UTC (rev 10657) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-24 09:58:06 UTC (rev 10658) @@ -477,7 +477,13 @@ def toHashCode(id): global classes global hashes + global jobconfig + cachePath = jobconfig["cachePath"] + + if not cachePath.endswith(os.sep): + cachePath += os.sep + try: hashes = cPickle.load(open(cachePath + "hashes", 'rb')) except (IOError, EOFError, cPickle.PickleError, cPickle.UnpicklingError): This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-25 09:48:33
|
Revision: 10671 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10671&view=rev Author: thron7 Date: 2007-10-25 02:48:31 -0700 (Thu, 25 Oct 2007) Log Message: ----------- Using toHashCode to construct file names for the cache (in getCompiled) and the resulting output file (in storeCompiledPackage). Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-25 08:32:48 UTC (rev 10670) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-25 09:48:31 UTC (rev 10671) @@ -564,6 +564,9 @@ variantsId = generateVariantCombinationId(variants) processId = generateProcessCombinationId(buildProcess) + variantsId = gen_hashcode.toHashCode(variantsId, hashes, jobconfig["cachePath"]) + processId = gen_hashcode.toHashCode(processId , hashes, jobconfig["cachePath"]) + packageFileName = packageFileName.replace("$variants", variantsId) packageFileName = packageFileName.replace("$process", processId) @@ -1054,6 +1057,9 @@ variantsId = generateVariantCombinationId(variants) processId = generateProcessCombinationId(process) + variantsId = gen_hashcode.toHashCode(variantsId, hashes, jobconfig["cachePath"]) + processId = gen_hashcode.toHashCode(processId , hashes, jobconfig["cachePath"]) + if variantsId != "": variantsId = "-" + variantsId This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-10-27 19:51:43
|
Revision: 10687 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10687&view=rev Author: wpbasti Date: 2007-10-27 12:51:41 -0700 (Sat, 27 Oct 2007) Log Message: ----------- Reduced verbosity Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-27 19:49:56 UTC (rev 10686) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-27 19:51:41 UTC (rev 10687) @@ -333,7 +333,7 @@ # Resolve modules/regexps - console.info(" - Resolving modules/regexps...") + console.debug(" - Resolving modules/regexps...") smartInclude = resolveComplexDefs(smartInclude) explicitInclude = resolveComplexDefs(explicitInclude) smartExclude = resolveComplexDefs(smartExclude) @@ -1382,8 +1382,8 @@ modules[mod].append(id) - console.info(" - Found %s modules" % len(modules)) - console.info("") + console.debug(" - Found %s modules" % len(modules)) + console.debug("") This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-10-27 20:03:49
|
Revision: 10690 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10690&view=rev Author: wpbasti Date: 2007-10-27 13:03:47 -0700 (Sat, 27 Oct 2007) Log Message: ----------- Minor changes Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-27 19:58:07 UTC (rev 10689) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-27 20:03:47 UTC (rev 10690) @@ -46,13 +46,13 @@ * Each part defines a part of the application which you want to load separately * A part could be of visual or logical nature * Each part may result into multiple packages (script files) -* The number of packages could be exponential to the number of views +* The number of packages could be exponential to the number of parts but through the optimization this is often not the case -* You can automatically collapse the important views. Such an important -view may be the initial application class (application layout frame) or +* You can automatically collapse the important parts. Such an important +part may be the initial application class (application layout frame) or the splashscreen. Collapsing reduces the number of packages for the -defined views. However collapsing badly influences the fine-grained nature -of the package system and should be ommitted for non-initial views normally. +defined parts. However collapsing badly influences the fine-grained nature +of the package system and should be ommitted for non-initial parts normally. * Further optimization includes support for auto-merging small packages. The relevant size to decide if a package is too small is the token size which is defined by the author of the job. The system calculates the token size of @@ -61,9 +61,9 @@ Internals ====================== * All merges happen from right to left when the package list is sorted by priority. -The main theory is that a package which is used by multiple views must have the dependencies +The main theory is that a package which is used by multiple parts must have the dependencies solved by both of them. So the merge will always happen into the next common package of -both views from the current position to the left side. +both parts from the current position to the left side. * There are some utility method which @@ -344,7 +344,7 @@ # - # PREPROCESS PHASE: VIEWS + # PREPROCESS PHASE: PARTS # if execMode == "parts": @@ -645,7 +645,7 @@ ###################################################################### -# VIEW/PACKAGE SUPPORT +# PART SUPPORT ###################################################################### def processParts(partClasses, partBits, includeDict, loadDeps, runDeps, variants, collapseParts, optimizeLatency, buildScript, buildProcess): This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-10-27 21:05:41
|
Revision: 10693 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10693&view=rev Author: wpbasti Date: 2007-10-27 14:05:40 -0700 (Sat, 27 Oct 2007) Log Message: ----------- Wording improvements Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-27 20:43:28 UTC (rev 10692) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-27 21:05:40 UTC (rev 10693) @@ -688,7 +688,7 @@ # Assign classes to packages console.debug("") - console.info(">>> Assigning classes to packages...") + console.debug(">>> Assigning classes to packages...") # References packageId -> class list packageClasses = {} @@ -722,7 +722,7 @@ # Assign packages to parts - console.info(">>> Assigning packages to parts...") + console.debug(">>> Assigning packages to parts...") partPackages = {} for partId in partClasses: @@ -821,9 +821,9 @@ console.debug("") - console.info(">>> Creating packages...") + console.info(">>> Compiling packages...") for packageId in sortedPackageIds: - console.info(" - Compiling package #%s:" % packageId, False) + console.info(" - Package #%s:" % packageId, False) packageFileName = "%s_%s" % (buildScript, packageId) packageSize = storeCompiledPackage(packageClasses[packageId], packageFileName, loadDeps, runDeps, variants, buildProcess, pos+1) This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-10-28 18:45:19
|
Revision: 10700 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10700&view=rev Author: wpbasti Date: 2007-10-28 11:45:17 -0700 (Sun, 28 Oct 2007) Log Message: ----------- Minor change Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-28 18:43:57 UTC (rev 10699) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-28 18:45:17 UTC (rev 10700) @@ -74,9 +74,6 @@ the classes of the current module * verbose{Boolean}: If verbose mode is enabled * quiet{Boolean}: If quiet mode is enabled - -* All cache data is automatically stored into "cache2"". The path is automatically - detected through the location of the generator script. """ import sys, re, os, optparse, math, cPickle, copy, sets, zlib This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-29 14:43:09
|
Revision: 10705 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10705&view=rev Author: thron7 Date: 2007-10-29 07:43:06 -0700 (Mon, 29 Oct 2007) Log Message: ----------- Improved logging and interface to storeSourceScript(). Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-29 14:21:33 UTC (rev 10704) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-29 14:43:06 UTC (rev 10705) @@ -442,19 +442,18 @@ if sourceScript != None: for packageId in pkgIds: - console.info("Generating source includer for package %s:" % packageId) - sourceText = storeSourceScript(pkg2classes[packageId], sourceScript+"-"+str(packageId), - variants, variantSetPos+1) + fileId = "%s-%s-%s.js" % (sourceScript, str(packageId), str(variantSetPos+1)) + console.info("Generating source includer for package %s: %s" % (packageId, fileId)) + sourceText = storeSourceScript(pkg2classes[packageId], fileId, variants) -def storeSourceScript(classList, packageFileName, variants, variantPos): +def storeSourceScript(classList, fileId, variants): global classes scriptBlocks = "" dict = _arrayToDict(classList) sortedClasses = deputil.sortClasses(dict, variants) - fileId = "%s-%s.js" % (packageFileName, variantPos) for f in sortedClasses: cEntry = classes[f] This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-29 17:29:18
|
Revision: 10707 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10707&view=rev Author: thron7 Date: 2007-10-29 10:29:07 -0700 (Mon, 29 Oct 2007) Log Message: ----------- Settings and variants support for source build. Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-29 15:17:24 UTC (rev 10706) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-29 17:29:07 UTC (rev 10707) @@ -87,6 +87,8 @@ from modules import filetool from modules import textutil from modules import simplejson +from modules import settings +from modules import variants as mvariants from generator2 import apidata from generator2 import cachesupport @@ -441,14 +443,32 @@ console.outdent() if sourceScript != None: + _settings = [] + _settings.append("qx.isSource:true") + _settings = settings.generate(_settings, True) + _variants = [] + _variants = mvariants.generate(_variants, True) for packageId in pkgIds: fileId = "%s-%s-%s.js" % (sourceScript, str(packageId), str(variantSetPos+1)) console.info("Generating source includer for package %s: %s" % (packageId, fileId)) - sourceText = storeSourceScript(pkg2classes[packageId], fileId, variants) + sourceText = getSourceIncludeList(pkg2classes[packageId], variants) + sourceText = sourceText.replace("'", "\\'") + sourceText = wrapInlineSource([_settings, _variants], False) + "\n" + sourceText + sourceText = "document.write('%s');" % sourceText + filetool.save(fileId, sourceText) -def storeSourceScript(classList, fileId, variants): +def wrapInlineSource(sourceArr, pNewLine): + if pNewLine: + nl = "\n" + else: + nl = "" + wrapped = '<script type="text/javascript">%s</script>%s' % (nl.join(sourceArr), nl) + return wrapped + + +def getSourceIncludeList(classList, variants): global classes scriptBlocks = "" @@ -471,12 +491,9 @@ scriptBlocks += '<script type="text/javascript" src="%s"></script>' % uri scriptBlocks += "\n" - sourceScript = "document.write('%s');" % scriptBlocks.replace("'", "\\'") + return scriptBlocks - filetool.save(fileId, sourceScript) - return sourceScript - def _arrayToDict(arr): dict = {} for e in arr: This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-30 10:11:44
|
Revision: 10710 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10710&view=rev Author: thron7 Date: 2007-10-30 03:11:42 -0700 (Tue, 30 Oct 2007) Log Message: ----------- Add settings and variant support for build versions (non-parts only). Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-30 10:09:26 UTC (rev 10709) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-30 10:11:42 UTC (rev 10710) @@ -88,7 +88,10 @@ from modules import textutil from modules import simplejson from modules import settings -from modules import variants as mvariants +from modules import variants as m_variants +from modules import tokenizer +#from modules import compiler as m_compiler +from modules import treegenerator from generator2 import apidata from generator2 import cachesupport @@ -423,8 +426,11 @@ if buildScript != None or sourceScript != None: + _settings = [] + _variants = [] if execMode == "parts": (pkgIds, pkg2classes, part2pkgs) = partutil.getPackages(partClasses, partBits, includeDict, variants, collapseParts, optimizeLatency) + pParts = True else: # simulate package @@ -432,22 +438,31 @@ pkg2classes = { "1" : includeDict.keys() } + pParts = False if buildScript != None: + _settings.append("qx.isSource:false") + _settings = settings.generate(_settings, True) + _variants = m_variants.generate(_variants, True) + prelude = "".join([_settings, _variants]) + compiledPrelude = compiler._compileClassHelper(treegenerator.createSyntaxTree(tokenizer.parseStream(prelude))) for packageId in pkgIds: console.info("Compiling classes for package %s:" % packageId, False) - packageSize = storeCompiledPackage(pkg2classes[packageId], buildScript, variants, buildProcess, variantSetPos+1) + packageSize, compiledContent = getCompiledPackage(pkg2classes[packageId], variants, buildProcess) + # Saving compiled content + fileId = "%s-%s" % (buildScript, variantSetPos+1) + if not pParts: # it's a simulated package + compiledContent = compiledPrelude + compiledContent + filetool.save(fileId + ".js", compiledContent) console.indent() console.debug("Done: %s" % packageSize) console.outdent() if sourceScript != None: - _settings = [] _settings.append("qx.isSource:true") _settings = settings.generate(_settings, True) - _variants = [] - _variants = mvariants.generate(_variants, True) + _variants = m_variants.generate(_variants, True) for packageId in pkgIds: fileId = "%s-%s-%s.js" % (sourceScript, str(packageId), str(variantSetPos+1)) console.info("Generating source includer for package %s: %s" % (packageId, fileId)) @@ -509,16 +524,13 @@ # COMMON COMPILED PKG SUPPORT ###################################################################### -def storeCompiledPackage(includeDict, packageFileName, variants, buildProcess, variantPos): - fileId = "%s-%s" % (packageFileName, variantPos) +def getCompiledPackage(includeDict, variants, buildProcess): # Compiling classes sortedClasses = deputil.sortClasses(includeDict, variants) compiledContent = compiler.compileClasses(sortedClasses, variants, buildProcess) - # Saving compiled content - filetool.save(fileId + ".js", compiledContent) - return getContentSize(compiledContent) + return getContentSize(compiledContent), compiledContent This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-30 10:50:39
|
Revision: 10714 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10714&view=rev Author: thron7 Date: 2007-10-30 03:50:28 -0700 (Tue, 30 Oct 2007) Log Message: ----------- Don't compile prelude (variants and settings). Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-30 10:36:01 UTC (rev 10713) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-30 10:50:28 UTC (rev 10714) @@ -445,14 +445,13 @@ _settings = settings.generate(_settings, True) _variants = m_variants.generate(_variants, True) prelude = "".join([_settings, _variants]) - compiledPrelude = compiler._compileClassHelper(treegenerator.createSyntaxTree(tokenizer.parseStream(prelude))) for packageId in pkgIds: console.info("Compiling classes for package %s:" % packageId, False) packageSize, compiledContent = getCompiledPackage(pkg2classes[packageId], variants, buildProcess) # Saving compiled content fileId = "%s-%s" % (buildScript, variantSetPos+1) if not pParts: # it's a simulated package - compiledContent = compiledPrelude + compiledContent + compiledContent = prelude + compiledContent filetool.save(fileId + ".js", compiledContent) console.indent() This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-30 11:40:28
|
Revision: 10715 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10715&view=rev Author: thron7 Date: 2007-10-30 04:40:09 -0700 (Tue, 30 Oct 2007) Log Message: ----------- Minor change. Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-30 10:50:28 UTC (rev 10714) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-30 11:40:09 UTC (rev 10715) @@ -89,9 +89,6 @@ from modules import simplejson from modules import settings from modules import variants as m_variants -from modules import tokenizer -#from modules import compiler as m_compiler -from modules import treegenerator from generator2 import apidata from generator2 import cachesupport This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <th...@us...> - 2007-10-30 15:07:48
|
Revision: 10719 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10719&view=rev Author: thron7 Date: 2007-10-30 07:03:36 -0700 (Tue, 30 Oct 2007) Log Message: ----------- Better variants support. Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-30 13:44:03 UTC (rev 10718) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-10-30 14:03:36 UTC (rev 10719) @@ -424,7 +424,7 @@ if buildScript != None or sourceScript != None: _settings = [] - _variants = [] + _variants = variantsToArray(variants) # need alternative format for variants.generate() if execMode == "parts": (pkgIds, pkg2classes, part2pkgs) = partutil.getPackages(partClasses, partBits, includeDict, variants, collapseParts, optimizeLatency) pParts = True @@ -469,7 +469,14 @@ filetool.save(fileId, sourceText) +def variantsToArray(variants): + varr = [] + for elem in variants: + varr.append(":".join([elem['id'], elem['value']])) + return varr + + def wrapInlineSource(sourceArr, pNewLine): if pNewLine: nl = "\n" This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-09 23:46:57
|
Revision: 10786 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10786&view=rev Author: wpbasti Date: 2007-11-09 15:46:53 -0800 (Fri, 09 Nov 2007) Log Message: ----------- Minor debug improvement Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-09 23:38:37 UTC (rev 10785) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-09 23:46:53 UTC (rev 10786) @@ -371,7 +371,8 @@ variantSets = variantsupport.computeCombinations(getJobConfig("variants", {})) for variantSetPos, variants in enumerate(variantSets): - console.head("PROCESSING VARIANT SET %s/%s" % (variantSetPos+1, len(variantSets))) + if len(variantSets) > 1: + console.head("PROCESSING VARIANT SET %s/%s" % (variantSetPos+1, len(variantSets))) # Debug variant combination This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-12 15:51:35
|
Revision: 10792 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10792&view=rev Author: wpbasti Date: 2007-11-12 07:51:33 -0800 (Mon, 12 Nov 2007) Log Message: ----------- Fixed part support... needs further polishing Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-12 15:30:27 UTC (rev 10791) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-12 15:51:33 UTC (rev 10792) @@ -247,8 +247,10 @@ + packageCfg = self.getConfig("packages") + # Use include/exclude - if not self.getConfig("packages"): + if not packageCfg: self.apiJob(include) self.sourceJob(include, variants, str(variantSetPos)) self.compileJob(include, variants, str(variantSetPos)) @@ -259,6 +261,10 @@ console.info("Preparing part configuration...") console.indent() + partsCfg = self.getConfig("packages/parts", []) + collapseCfg = self.getConfig("packages/collapse", []) + latencyCfg = self.getConfig("packages/latency", 0) + # Build bitmask ids for parts console.debug("Assigning bits to parts...") @@ -266,7 +272,7 @@ console.indent() partBits = {} partPos = 0 - for partId in userParts: + for partId in partsCfg: partBit = 1<<partPos console.debug("Part #%s => %s" % (partId, partBit)) @@ -279,22 +285,27 @@ # Resolving modules/regexps console.debug("Resolving part modules/regexps...") partClasses = {} - for partId in userParts: - partClasses[partId] = resolveComplexDefs(userParts[partId]) + for partId in partsCfg: + partClasses[partId] = self._resolveComplexDefs(partsCfg[partId]) console.outdent() - (pkgIds, pkg2classes, part2pkgs) = partutil.getPackages(partClasses, partBits, includeDict, variants, collapseParts, optimizeLatency) + (pkgIds, pkg2classes, part2pkgs) = self._partutil.getPackages(partClasses, partBits, include, variants, collapseCfg, latencyCfg) for pkgId in pkgIds: - sortedInclude = self._deputil.sortClasses(pkg2classes[pkgId], variants) + partInclude = self._deputil.sortClasses(pkg2classes[pkgId], variants) - self.sourceJob(sortedInclude, variants, str(variantSetPos), pkgId) - self.compileJob(sortedInclude, variants, str(variantSetPos), pkgId) + console.info("Creating package %s (%s classes)" % (pkgId, len(partInclude))) + console.indent() + self.sourceJob(partInclude, variants, str(variantSetPos), str(pkgId)) + self.compileJob(partInclude, variants, str(variantSetPos), str(pkgId)) + console.outdent() + + def cleanJob(self): cleanCfg = self.getConfig("clean") This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-19 10:26:31
|
Revision: 10808 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10808&view=rev Author: wpbasti Date: 2007-11-19 02:26:28 -0800 (Mon, 19 Nov 2007) Log Message: ----------- minor fix Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-19 08:54:48 UTC (rev 10807) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-19 10:26:28 UTC (rev 10808) @@ -193,7 +193,7 @@ # CORE: GENERATORS ###################################################################### -class Generator(): +class Generator: def __init__(self, config, console): self._config = config self._console = console This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-22 08:36:36
|
Revision: 10819 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10819&view=rev Author: wpbasti Date: 2007-11-22 00:36:33 -0800 (Thu, 22 Nov 2007) Log Message: ----------- Minor fix Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-22 08:29:24 UTC (rev 10818) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-22 08:36:33 UTC (rev 10819) @@ -239,7 +239,7 @@ self.runClean() # API Data Task - self.runApiData() + # self.runApiData() # Check for package configuration @@ -255,7 +255,7 @@ collapseCfg.append(bootPart) # Expanding expressions - self._console.debug("Resolving part regexps...") + self._console.debug("Expanding include expressions...") partIncludes = {} for partId in partsCfg: partIncludes[partId] = self._expandRegExps(partsCfg[partId]) @@ -324,7 +324,7 @@ - def apiJob(self, include): + def runApiData(self, include): apiPath = self._config.get("api/path") if not apiPath: This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-22 08:42:54
|
Revision: 10820 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10820&view=rev Author: wpbasti Date: 2007-11-22 00:42:50 -0800 (Thu, 22 Nov 2007) Log Message: ----------- Minor change Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-22 08:36:33 UTC (rev 10819) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-22 08:42:50 UTC (rev 10820) @@ -235,11 +235,7 @@ self._console.outdent() - # Cleanup Task - self.runClean() - # API Data Task - # self.runApiData() # Check for package configuration @@ -270,6 +266,12 @@ classList = self._deputil.getClassList(smartInclude, smartExclude, explicitInclude, explicitExclude, variants) self._console.outdent() + # Cleanup Task + self.runClean(classList) + + # API Data Task + self.runApiData(classList) + # Emulate configuration bootPart = "boot" @@ -287,7 +289,7 @@ - def runClean(self): + def runClean(self, classList): cleanCfg = self._config.get("clean") if not cleanCfg: @@ -324,13 +326,13 @@ - def runApiData(self, include): + def runApiData(self, classList): apiPath = self._config.get("api/path") if not apiPath: return - self._apiutil.storeApi(include, apiPath) + self._apiutil.storeApi(classList, apiPath) This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-22 09:23:04
|
Revision: 10821 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10821&view=rev Author: wpbasti Date: 2007-11-22 01:23:01 -0800 (Thu, 22 Nov 2007) Log Message: ----------- Added debug part task Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-22 08:42:50 UTC (rev 10820) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-22 09:23:01 UTC (rev 10821) @@ -286,9 +286,12 @@ # Compiled Task self.runCompiled(partToPackages, packageContent, bootPart, variants) + # Dependeny Debug Task + self.runDependencyDebug(partToPackages, packageContent, variants) + def runClean(self, classList): cleanCfg = self._config.get("clean") @@ -336,7 +339,37 @@ + def runDependencyDebug(self, partToPackages, packageContents, variants): + if not self._config.get("debug/dependencies", False): + return + self._console.info("Dependency debugging...") + self._console.indent() + + for packageId, packageContent in enumerate(packageContents): + self._console.info("Package %s" % packageId) + self._console.indent() + for classId in packageContent: + self._console.debug("Class: %s" % classId) + self._console.indent() + + for otherClassId in packageContent: + otherClassDeps = self._deputil.getDeps(otherClassId, variants) + + if classId in otherClassDeps["load"]: + self._console.debug("Used by: %s (load)" % otherClassId) + + if classId in otherClassDeps["run"]: + self._console.debug("Used by: %s (run)" % otherClassId) + + self._console.outdent() + self._console.outdent() + + self._console.outdent() + + + + def runCompiled(self, partToPackages, packageContents, bootPart, variants): if not self._config.get("compile/file"): return This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-22 09:40:19
|
Revision: 10822 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10822&view=rev Author: wpbasti Date: 2007-11-22 01:40:07 -0800 (Thu, 22 Nov 2007) Log Message: ----------- Minor addition Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-22 09:23:01 UTC (rev 10821) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-22 09:40:07 UTC (rev 10822) @@ -349,6 +349,11 @@ for packageId, packageContent in enumerate(packageContents): self._console.info("Package %s" % packageId) self._console.indent() + + for partId in partToPackages: + if packageId in partToPackages[partId]: + self._console.info("Part %s" % partId) + for classId in packageContent: self._console.debug("Class: %s" % classId) self._console.indent() This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-26 14:28:35
|
Revision: 10825 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10825&view=rev Author: wpbasti Date: 2007-11-26 06:26:02 -0800 (Mon, 26 Nov 2007) Log Message: ----------- Better error handling Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-22 15:19:51 UTC (rev 10824) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-26 14:26:02 UTC (rev 10825) @@ -681,25 +681,35 @@ def _expandRegExps(self, entries): - classes = self._classes + result = [] - content = [] - for entry in entries: - regexp = textutil.toRegExp(entry) + # Fast path: Try if a matching class could directly be found + if entry in self._classes: + result.append(entry) - for className in classes: - if regexp.search(className): - if not className in content: - content.append(className) + else: + regexp = textutil.toRegExp(entry) + expanded = [] - return content + for classId in self._classes: + if regexp.search(classId): + if not classId in content: + result.append(classId) + if len(expanded) == 0: + self._console.error("Expression gives no results. Maybe malformed expression?: %s" % entry) + sys.exit(1) + result.extend(expanded) + return result + + + ###################################################################### # UTIL ###################################################################### This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-27 19:17:12
|
Revision: 10831 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10831&view=rev Author: wpbasti Date: 2007-11-27 11:17:11 -0800 (Tue, 27 Nov 2007) Log Message: ----------- Added support for runtime variants and settings. Added support for feature sets in external variant and settings files. Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-27 19:16:10 UTC (rev 10830) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-27 19:17:11 UTC (rev 10831) @@ -105,6 +105,12 @@ parser.add_option("-v", "--verbose", action="store_true", dest="verbose", default=False, help="Verbose output mode (Extra verbose).") parser.add_option("-l", "--logfile", dest="logfile", metavar="FILENAME", default="", type="string", help="Log file") + # runtime addons + parser.add_option("--use-setting", action="extend", dest="usedSettings", metavar="KEY:VALUE", type="string", default=[], help="Used settings") + parser.add_option("--select-variant", action="extend", dest="selectedVariants", metavar="KEY:VALUE", type="string", default=[], help="Selected variants") + parser.add_option("--load-variants", action="extend", dest="variantFiles", metavar="NAMESPACE:FILE", type="string", default=[], help="Variant files to load") + parser.add_option("--load-settings", action="extend", dest="settingFiles", metavar="NAMESPACE:FILE", type="string", default=[], help="Setting files to load") + if len(sys.argv[1:]) == 0: basename = os.path.basename(sys.argv[0]) print "usage: %s [options]" % basename @@ -142,12 +148,94 @@ # Convert into Config class instance config = configsupport.Config(config) + + # Preprocess selected variants + variants = {} + for entry in options.selectedVariants: + splitted = entry.split(":") + variants[splitted[0]] = splitted[1] + + + # Preprocess variant files + for entry in options.variantFiles: + splitted = entry.split(":") + namespace = splitted[0] + filename = splitted[1] + + runtime = {} + for key in variants: + runtime[key[len(namespace + "."):]] = variants[key] + + execfile(filename, {}, runtime) + + for key in runtime: + if key == "" or key.startswith("_") or key.isupper(): + continue + + namespaced = "%s.%s" % (namespace, key) + + if variants.has_key(namespaced) and variants[namespaced] != runtime[key]: + console.error("Overwriting key not allowed: %s" % key) + sys.exit(1) + + value = runtime[key] + + if value == True: + value = "on" + elif value == False: + value = "off" + + variants[namespaced] = value + + + # Preprocess used settings + settings = {} + for entry in options.usedSettings: + splitted = entry.split(":") + settings[splitted[0]] = splitted[1] + + + # Preprocess setting files + for entry in options.settingFiles: + splitted = entry.split(":") + namespace = splitted[0] + filename = splitted[1] + + runtime = {} + + runtime = {} + for key in variants: + runtime[key[len(namespace + "."):]] = variants[key] + + execfile(filename, {}, runtime) + + for key in runtime: + if key == "" or key.startswith("_") or key.isupper(): + continue + + namespaced = "%s.%s" % (namespace, key) + + # Settings must be unique + if variants.has_key(namespaced): + continue + + value = runtime[key] + + if value == True: + value = "on" + elif value == False: + value = "off" + + settings[namespaced] = value + + # Processing jobs... for job in options.jobs: console.head("Executing: %s" % job, True) - Generator(config.split(job), console) + generator = Generator(config.split(job), console, variants, settings) + def resolve(console, config, jobs): console.info("Resolving jobs...") console.indent() @@ -158,6 +246,7 @@ console.outdent() + def resolveEntry(console, config, job): if not config.has_key(job): console.warn("No such job: %s" % job) @@ -178,6 +267,7 @@ data["resolved"] = True + def mergeEntry(target, source): for key in source: if not target.has_key(key): @@ -194,9 +284,11 @@ ###################################################################### class Generator: - def __init__(self, config, console): + def __init__(self, config, console, variants, settings): self._config = config self._console = console + self._variants = variants + self._settings = settings self._cache = cachesupport.Cache(self._config.get("cache/path"), self._console) self._classes = classpath.getClasses(self._config.split("library"), self._console) @@ -217,8 +309,7 @@ smartExclude, explicitExclude = self.getExcludes() # Processing all combinations of variants - variantData = self._config.get("variants", {}) - variantSets = variantsupport.computeCombinations(variantData) + variantSets = variantsupport.computeCombinations(self.getVariants()) # Iterate through variant sets for variantSetNum, variants in enumerate(variantSets): @@ -227,12 +318,11 @@ # Debug variant combination - if len(variants) > 0: - self._console.debug("Selected variants:") - self._console.indent() - for key in variants: - self._console.debug("%s = %s" % (key, variants[key])) - self._console.outdent() + self._console.debug("Selected variants:") + self._console.indent() + for key in variants: + self._console.debug("%s = %s" % (key, variants[key])) + self._console.outdent() @@ -328,7 +418,36 @@ return + def getSettings(self): + settings = {} + settingsConfig = self._config.get("settings", {}) + settingsRuntime = self._settings + for key in settingsConfig: + settings[key] = settingsConfig[key] + + for key in settingsRuntime: + settings[key] = settingsRuntime[key] + + return settings + + + + def getVariants(self): + variants = {} + variantsConfig = self._config.get("variants", {}) + variantsRuntime = self._variants + + for key in variantsConfig: + variants[key] = variantsConfig[key] + + for key in variantsRuntime: + variants[key] = [variantsRuntime[key]] + + return variants + + + def runApiData(self, classList): apiPath = self._config.get("api/path") @@ -392,10 +511,9 @@ format = self._config.get("compile/format", False) # Read in settings - settings = self._config.get("settings", {}) + settings = self.getSettings() - # Generating boot script self._console.info("Generating boot script...") @@ -463,7 +581,7 @@ format = self._config.get("source/format", False) # Read in settings - settings = self._config.get("settings", {}) + settings = self.getSettings() # Add data from settings, variants and packages sourceBlocks = [] This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <wp...@us...> - 2007-11-27 19:25:20
|
Revision: 10832 http://qooxdoo.svn.sourceforge.net/qooxdoo/?rev=10832&view=rev Author: wpbasti Date: 2007-11-27 11:25:17 -0800 (Tue, 27 Nov 2007) Log Message: ----------- Minor change Modified Paths: -------------- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py Modified: branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py =================================================================== --- branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-27 19:17:11 UTC (rev 10831) +++ branches/legacy_0_7_x/qooxdoo/frontend/framework/tool/generator2.py 2007-11-27 19:25:17 UTC (rev 10832) @@ -325,9 +325,6 @@ self._console.outdent() - - - # Check for package configuration if self._config.get("packages"): # Reading configuration This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |