Thread: [Pcbsd-pbi-dev] A new library handling idea
Status: Beta
Brought to you by:
kmoore134
Menu
▾
▴
pcbsd-pbi-dev
|
From: Federico L. <flo...@gm...> - 2006-02-11 07:13:42
|
Ok, heres a brand new idea on handaling libraries:
Directory structure:
PBI Root
-> ProgLib
-> libfoo.so.1
-> DepLib
-> libc.so.6
-> libqt.so.2
-> libkde.so.8
-> lib
<empty>
We have a PBI called FooApp, which needs libc.so.6 libqt.so.2 libkde.so.8an=
d
libfoo.so.1
Now, libfoo.so.1 is a program specific library, NOT a normal dependancy, (i=
t
comes with FooApp),
so it is stored in ProgLib, while the other libs are stored in DepLib....
Now heres the killer bit...
The lib path of the program will be set to the lib dir, but since its empty
a few things will happen first...
A script (which i'm working on) will:
* Symlink all the libraries in the ProgLib folder to the lib folder (as the=
y
wont beable to be found anywhere else)
^ Generate an md5 sum of all the libraries in the DepLib folder and save it
as DepLib/md5 (once off)
* Compare those md5's to that of the systems libraries, if theres a match,
the system ones get symlinked into the
lib folder, but if theres no match, the DepLib ones get symlinked in
Please tell me what you think of this idea, as it should elivate the proble=
m
of wasted memory, while still allowing
things that dont have all the required libs in the system / the ones in the
system are too out of date / too new to
still work!
PS. The DepLib and ProgLib folders tie in with my ports-pbi compatability
plan ;)
Cheers
On 2/10/06, Federico Lorenzi <flo...@gm...> wrote:
>
> I'm not a programmer or anything, but i always thought that if lets say
> FooApp loaded libfoo.so.1 into memory, as it needed it, and BarAPP
> needed the same lib, it would be smart enough too see that its already
> loaded, and simply use it.
> Then again i'm probably wrong :) Although that would be the idea
> situation.
>
> Cheers
>
> On 2/10/06, Richy T < bio...@gm...> wrote:
> >
> > Hello pbi developers!
> > I have been doing some reading on our pcbsd forums.
> > There is some really good information there about our format.
> > I really think all PBI developers should read this.
> > The information posted on the forum might really help make
> > .PBI truly a great binary format for pcbsd and maybe freebsd!
> > Here is the post. -->
> >
> > From: Almindor
> > > ---------------
> > > I'm a programmer so I'll post my view on PBIs and other packaging
> > > here.
> > >
> > > PBI is a .app style packaging system with "all in one" even basic
> > > dependencies. The good of this is that it's simple, easy and has
> > > (theoreticly, in practice this isn't true) no conflicts.
> > >
> > > Packaging ala Ports/Debs/RPMs is a complicated (altho not always for
> > > the end user especialy if they have good net connection, debs are rea=
l good
> > > here) process which required dependency checks etc.
> > >
> > > Now for the pros and cons from my perspective:
> > >
> > > PBIs are great for programmers/developers since they make it easy for
> > > them to distribute their apps without much fuzz.
> > > PBIs however have 2 major problems as-is, one is directly tied to the
> > > FreeBSD system underneath:
> > >
> > > 1. It conflicts with ports/packages
> > > 2. It copies all libs for all apps and makes ALOT of duality with .so
> > > files.
> > >
> > > Now #2 seems like no problem to most of you guys but let me elaborate=
.
> > >
> > > Dynamic libraries, or shared objects as Unix world calls them are
> > > pieces of code (and sometimes data too) which are saved in a special =
way so
> > > that programs(apps) can "link" themselves to the code and use it.
> > >
> > > The whole concept of shared objects is, sharing memory Smile (how
> > > unexpected)
> > >
> > > What's wrong with per-program copy of .so files then?
> > >
> > > Memory usage. Start up KDE and a few apps, then look at your memory
> > > usage via some monitor program. Notice that most apps (visual kde etc=
) take
> > > 10+megs of RAM. This ISNT REAL MEMORY.
> > >
> > > Because all of these programs use more or less the same .so files, al=
l
> > > this memory is shared.
> > >
> > > Now imagine a system where virtualy everything comes from PBI. Every
> > > program would really hog ram with it's own copies of .so files (and h=
ence
> > > their existance is a hindrance, since smart-static linking would prod=
uce
> > > much smaller footprint, less conflicts etc.)
> > >
> > > Conclusion? It's a good idea on a bad system. I've nothing agains Uni=
x
> > > in general but Unix world (and windows too to an extent) decided to g=
o the
> > > "shared object" path. Every unix binary today links dynamicly etc.
> > >
> > > Is there a different approach?
> > >
> > > Well there is, but if it's better is a nice question. Now before I
> > > start this please note that I AM biased Smile
> > >
> > > Free Pascal compiler can produce "smart linked" binaries. These
> > > binaries are static but contrary to popular belief take very little s=
pace on
> > > disk and in memory. (Hello world app in C static linked is ~800kb!! i=
n FPC
> > > smartlinked it's ~20kb on disk, I'll explain why)
> > >
> > > Smart linking is a technique which "links in" only used code from a
> > > library.
> > >
> > > So let's say libC is smartlinkable (it's not btw).
> > >
> > > If you only used printf() your binary would only grow "so" much.
> > >
> > > If you use dynamic .so, your binary will remain small on disk, but
> > > will load the whole libC into memory because of printf(). Now imagine=
all
> > > programs loading whole libC into memory per-copy. This is what is don=
e with
> > > PBIs and that's the bad idea. PBIs would work wonders on a "smartlink=
" based
> > > system where only real basic system libs would be dynamic, and all ot=
her dev
> > > libs would be smartlinkable. This way programs would have minimal foo=
tprints
> > > and no conflicts would happen (they would also have a tendency to wor=
k
> > > longer since ABI is of no concern).
> > >
> > > Hope this all makes a bit of sense Smile
> > >
> > > Ales
> >
> >
> >
> > POST REPLY:
> >
> > From: pcbsdusr
> > > ----------------
> > > I have been thinking of this as well and i am in the process of
> > > writing a hibrid concept which uses programs in their own folders but=
shares
> > > dependencies while maintaining the ability to use multiple versions o=
f any
> > > given library side by side.
> > >
> > > no multiplication of libraries =3D No wasted disk space and no memory
> > > waste while multitasking with apps using common dependencies.
> > >
> > > of course, i am writing this for apps only... There will be always
> > > duplication until we maintain freebsd intact under pcbsd...
> >
> >
> > I really hope you guys can update the format to work this way.
> > It would make the .pbi format truly one of, if not the best format for
> > pcbsd/freebsd/UNIX systems.
> > Hope this gives you guys some insight and inspiration on how to update
> > the .pbi format.
> >
> > - Richy
> >
> >
> >
> >
> >
> >
>
|
|
From: Taras D. <go...@gm...> - 2006-02-13 18:09:57
|
Reading your message I got a few questions:
Lets imagine situation when user installs 2+ programs (from pbi), and
they use the same shared library. According to your algorythm the
first of those programms will not find a match of its specific
library in the /lib/ folder and will create a symlink to it there.
Then second prog during its installiation will find a match of its
specific library in the /lib/ folder and will not change anything. And
so on.
Now lets imagine that user uninstalls the first app - will the
uninstall also remove the lib? Or the system will keep it forever
regardless of what progs are installed? Or maybe the system will check
if there is perhaps 1 app that refers to this lib - and if this is
FALSE - it will remove the lib? If so - it will be very similar to the
FreeBSD's dependency list...
As I remember Kris wrote his thoughts about shared libraries at the
early days of PCBSD... He said that the advantages of the pbi (user
doesnt care about dependencies when installs a programm) are surpass
the disadvantages of multiple loading of shared objects... So its not
an unexpected news about this...
Actualy as a programmer i can say that when you staticaly linking the
programm against some library using C/C++ compiller - the compiller
extracts only needed routines from the lib to add them to the
executable (similiar to the behavior, explained for the pascal
compiller by the guy before me ). But if we need a shared compiling -
there is also a way to decrease the shared object's size up to 1/3 of
its basic size - there are utilities called strippers for this purpose
( dont recall the exact names right now ).
By the way it will be useful to check the information of how another
OSes fixing this problem. For example Windows OS uses the same model
as pbi. Microsoft suggests third party developers to place their
shared libraries into their own folders but not to the global dirs
like system32 etc. It causes the situation with multiple loading of
the same libs, but it simplifies the developing of user apps for
developers... Eating the RAM it however causes the rapid growing of
number of applications available for this platform...
IMHO the main dilemma of the pbi isnt a multiple loading of the same
libs, but is how to combine usual dependency system with pbi - in
other words - avoiding of the multiple installing of same
sub-tools(quick PBI example - apps "A" & "B" require app "C" as their
subsystem - "A" will install "C" to its subfolder or somewhere else,
"B" does the same. As a result "C" installed twice)
If the information about programms installed via pbi system will be
added to the pkg list and newly installed programms will refer to them
as on dependecies - we'll lose the idea of pbi. If we will create a
kind of windows/macOS registry - it will start the complete separation
from original FreeBSD logic...
We'll really need a good software architector for this - because it is
hard to add something new to the existing schemes. ;-)
Summarizing my long message i think it will be interesting to get
information from Mac users/coders on how does the MacOS act in same
situations since it is BSD based and also uses some of the Windows'
technics.
And also i think it will be cool to create a kind of official document
describing what to consider as specific or as global library / tool
when the developer creates its pbi or another soft for PCBSD (for
example should i consider required libc.so.6 as specific to my pbi
when the system has only libc.so.5 etc). Dont know how real is this
but it seems quite useful as for developer...
Regards, Taras aka Gorthaur
On 2/11/06, Federico Lorenzi <flo...@gm...> wrote:
> Ok, heres a brand new idea on handaling libraries:
>
> Directory structure:
> PBI Root
> -> ProgLib
> -> libfoo.so.1
> -> DepLib
> -> libc.so.6
> -> libqt.so.2
> -> libkde.so.8
> -> lib
> <empty>
>
> We have a PBI called FooApp, which needs libc.so.6 libqt.so.2 libkde.so.8
> and libfoo.so.1
> Now, libfoo.so.1 is a program specific library, NOT a normal dependancy, =
(it
> comes with FooApp),
> so it is stored in ProgLib, while the other libs are stored in DepLib....
> Now heres the killer bit...
>
> The lib path of the program will be set to the lib dir, but since its emp=
ty
> a few things will happen first...
> A script (which i'm working on) will:
> * Symlink all the libraries in the ProgLib folder to the lib folder (as t=
hey
> wont beable to be found anywhere else)
> ^ Generate an md5 sum of all the libraries in the DepLib folder and save =
it
> as DepLib/md5 (once off)
> * Compare those md5's to that of the systems libraries, if theres a match=
,
> the system ones get symlinked into the
> lib folder, but if theres no match, the DepLib ones get symlinked in
>
> Please tell me what you think of this idea, as it should elivate the prob=
lem
> of wasted memory, while still allowing
> things that dont have all the required libs in the system / the ones in t=
he
> system are too out of date / too new to
> still work!
>
> PS. The DepLib and ProgLib folders tie in with my ports-pbi compatability
> plan ;)
> Cheers
--
contact me:
email: go...@gm...
com...@us...
icq: 166956956
mobile: +380969474990 (djuice)
|
|
From: Federico L. <flo...@gm...> - 2006-02-14 14:10:07
|
NOO... you've got me all wrong :) EACH PBI will have a lib folder, for example /Programs/FooApp1.0/lib and /Programs/BarApp1.0/lib then when each program runs it will automatically clear the folder and create the symlinks. Sorry i can= t reply in more details but i'm a bit stripped for time now. Cheers On 2/13/06, Taras Danko <go...@gm...> wrote: > > Reading your message I got a few questions: > Lets imagine situation when user installs 2+ programs (from pbi), and > they use the same shared library. According to your algorythm the > first of those programms will not find a match of its specific > library in the /lib/ folder and will create a symlink to it there. > Then second prog during its installiation will find a match of its > specific library in the /lib/ folder and will not change anything. And > so on. > > Now lets imagine that user uninstalls the first app - will the > uninstall also remove the lib? Or the system will keep it forever > regardless of what progs are installed? Or maybe the system will check > if there is perhaps 1 app that refers to this lib - and if this is > FALSE - it will remove the lib? If so - it will be very similar to the > FreeBSD's dependency list... > > As I remember Kris wrote his thoughts about shared libraries at the > early days of PCBSD... He said that the advantages of the pbi (user > doesnt care about dependencies when installs a programm) are surpass > the disadvantages of multiple loading of shared objects... So its not > an unexpected news about this... > > Actualy as a programmer i can say that when you staticaly linking the > programm against some library using C/C++ compiller - the compiller > extracts only needed routines from the lib to add them to the > executable (similiar to the behavior, explained for the pascal > compiller by the guy before me ). But if we need a shared compiling - > there is also a way to decrease the shared object's size up to 1/3 of > its basic size - there are utilities called strippers for this purpose > ( dont recall the exact names right now ). > > By the way it will be useful to check the information of how another > OSes fixing this problem. For example Windows OS uses the same model > as pbi. Microsoft suggests third party developers to place their > shared libraries into their own folders but not to the global dirs > like system32 etc. It causes the situation with multiple loading of > the same libs, but it simplifies the developing of user apps for > developers... Eating the RAM it however causes the rapid growing of > number of applications available for this platform... > > IMHO the main dilemma of the pbi isnt a multiple loading of the same > libs, but is how to combine usual dependency system with pbi - in > other words - avoiding of the multiple installing of same > sub-tools(quick PBI example - apps "A" & "B" require app "C" as their > subsystem - "A" will install "C" to its subfolder or somewhere else, > "B" does the same. As a result "C" installed twice) > > If the information about programms installed via pbi system will be > added to the pkg list and newly installed programms will refer to them > as on dependecies - we'll lose the idea of pbi. If we will create a > kind of windows/macOS registry - it will start the complete separation > from original FreeBSD logic... > We'll really need a good software architector for this - because it is > hard to add something new to the existing schemes. ;-) > > Summarizing my long message i think it will be interesting to get > information from Mac users/coders on how does the MacOS act in same > situations since it is BSD based and also uses some of the Windows' > technics. > And also i think it will be cool to create a kind of official document > describing what to consider as specific or as global library / tool > when the developer creates its pbi or another soft for PCBSD (for > example should i consider required libc.so.6 as specific to my pbi > when the system has only libc.so.5 etc). Dont know how real is this > but it seems quite useful as for developer... > > > Regards, Taras aka Gorthaur > > > On 2/11/06, Federico Lorenzi <flo...@gm...> wrote: > > Ok, heres a brand new idea on handaling libraries: > > > > Directory structure: > > PBI Root > > -> ProgLib > > -> libfoo.so.1 > > -> DepLib > > -> libc.so.6 > > -> libqt.so.2 > > -> libkde.so.8 > > -> lib > > <empty> > > > > We have a PBI called FooApp, which needs libc.so.6 libqt.so.2 > libkde.so.8 > > and libfoo.so.1 > > Now, libfoo.so.1 is a program specific library, NOT a normal dependancy= , > (it > > comes with FooApp), > > so it is stored in ProgLib, while the other libs are stored in > DepLib.... > > Now heres the killer bit... > > > > The lib path of the program will be set to the lib dir, but since its > empty > > a few things will happen first... > > A script (which i'm working on) will: > > * Symlink all the libraries in the ProgLib folder to the lib folder (as > they > > wont beable to be found anywhere else) > > ^ Generate an md5 sum of all the libraries in the DepLib folder and sav= e > it > > as DepLib/md5 (once off) > > * Compare those md5's to that of the systems libraries, if theres a > match, > > the system ones get symlinked into the > > lib folder, but if theres no match, the DepLib ones get symlinked in > > > > Please tell me what you think of this idea, as it should elivate the > problem > > of wasted memory, while still allowing > > things that dont have all the required libs in the system / the ones in > the > > system are too out of date / too new to > > still work! > > > > PS. The DepLib and ProgLib folders tie in with my ports-pbi > compatability > > plan ;) > > Cheers > > > -- > contact me: > email: go...@gm... > com...@us... > icq: 166956956 > mobile: +380969474990 (djuice) > |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X