Stuff that has not yet gone into the official build.
Post a reply

Re: Sid, Refracta and Refracta-tools

Sat Oct 05, 2013 9:47 pm

http://docs.python.org/2/tutorial/modules.html (section 6.1.3)

A program doesn’t run any faster when it is read from a .pyc or .pyo file than when it is read from a .py file; the only thing that’s faster about .pyc or .pyo files is the speed with which they are loaded.

On an installation, after deletion and reboot I observed their return. Example: wicd (uses python modules, see /usr/lib/python2.7/dist-packages/wicd) runs here on boot. The UI opened quick and performed normally.

But like I said, I'm just looking at ways of getting the bloat down, I don't actually understand python and might be wrong..

However ~15 MB was shaved off the "live" chroot, that just tipped the balance to get it on cd without purging apps.

One thing I forgot.. update Refracta scripts from git! so another update soon..

Re: Sid, Refracta and Refracta-tools

Sun Oct 06, 2013 2:45 pm

ISO now down to 682 MB (as reported by genisoimage and df -h)..

Code:
[sid: /]# apt-get update

## search installed packages not in configured repos

[sid: /]# aptitude search "~o"
i   epdfview                        - Lightweight pdf viewer based on poppler li
i   firmware-iwlwifi                - Binary firmware for Intel Wireless cards 
i   fuse-utils                      - Filesystem in Userspace (transitional pack
i A libmozjs10d                     - Mozilla SpiderMonkey JavaScript library   
i   live-hooks-refracta             - Additional Configuration Scripts for Debia
i   ntfsprogs                       - read/write NTFS driver for FUSE (transitio
i   python-central                  - register and build utility for Python pack
i   python2.6                       - Interactive high-level object-oriented lan
i   python2.6-minimal               - Minimal subset of the Python language (ver
i   refracta2usb                    - Make live-USB from cdrom iso or from runni
i   refractainstaller-base          - tool to install a running live-CD to hard
i   refractainstaller-gui           - tool to install a running live-CD to hard
i   refractasnapshot-base           - tool to create a live-CD from the running
i   refractasnapshot-gui            - tool to create a live-CD from the running
i   taskfer                         - todo list manager written in bash.       
i   yad                             - Yet Another Dialog - dialogs for shell   
i   youtube-viewer-refracta         - search and watch YouTube videos with MPlay
[sid: /]#

## SNIP

[sid: /]# apt-get purge python2.6* gcc-4.6*

# SNIP

The following packages will be REMOVED:
  cpp-4.6* gcc-4.6* gcc-4.6-base* python2.6* python2.6-minimal*
0 upgraded, 0 newly installed, 5 to remove and 4 not upgraded.
After this operation, 39.4 MB disk space will be freed.

## SNIP

[sid: /]# apt-get purge libmozjs10d python-central ntfsprogs fuse-utils
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  fuse-utils* libmozjs10d* ntfsprogs* python-central*
0 upgraded, 0 newly installed, 4 to remove and 4 not upgraded.
After this operation, 4,094 kB disk space will be freed.

## SNIP

[sid: /]# apt-get autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  libffi5
0 upgraded, 0 newly installed, 1 to remove and 4 not upgraded.
After this operation, 77.8 kB disk space will be freed.

## SNIP

[sid: /]# deborphan
[sid: /]#

If it's not in sid, and we haven't installed it manually, it shouldn't be there.. and what's gcc4.6 in sid for other than special case use?

Re the package list diff: Was hoping it was from a sid build but it looks like wheezy. It's what's come in and gone out as well as version changes plus my own bloat-aware changes (some of which now matter less, we got 18 MB of squashfs spare now)

Re: Sid, Refracta and Refracta-tools

Sun Oct 06, 2013 9:22 pm

dzz, thanks for testing the absence of pyc files and reporting the result !
I booted your "sid" iso to a live session and found that you can quickly eliminate AT LEAST 275Mb ("size on disk") worth of bloat.
I'm curious to hear what the reduction in squashfs/iso size will be.

The primarly source of bloat is from extraneous locale-specific files.
Yes, you already knew this (have installed/run bleachbit) but, evident in the iso, plenty of locales files are STILL present.
Boot your iso to a live session and "walk along with me" thru these steps:

boot using these params:
config=locales locale=en_GB.UTF-8

login as root/root

$ dpkg-reconfigure locales
and deselect any locales you'll not use
(FWIW, in testing I preserved: en, en_US, en_GB)

$ apt-get install localepurge
and page-down thru the listed locales.
Shouldn't need to make any changes, but then again, you might notice
that you had mis-marked a few during your "dpkg-reconfigure locales" run.
Respond "yes" to the "blahblah dpkg paths" prompt during install.
(If you respond NO... dog will kill a kitten)

Next, launch bleachbit, instructing it to delete localizations.
For the moment, have bb delete ONLY localizations. Leave everything else unticked.
In your previous use (prior to remastering), removing locale bloat via bleachbit was semi-futile
because, with each package install/update operation, dpkg was repopulating the locale-specific bloat.

observe the immediate cleanup result, as reported by bb.
Disk space recovered: 277.5Mb
Files deleted: 11344

The result when performed from an installed session (hard drive) should be even greater;
in live session, we cannot clean /usr/share/help (apparently it's path is a ro bindmount).

Re: Sid, Refracta and Refracta-tools

Sun Oct 06, 2013 11:19 pm

Because bleachbit + localepurge have some "blind spots", additional cleanup (recurring, and prior to each remastering operation) is necessary to weed out all extraneous locale files. If many package "update" or "install" operations have occurred since previous squashfs creation, the accumulated stray locale-specific bloat may consume 60Mb+ on disk. This represents an ongoing concern in live+persistence scenario, and is what led me to pursue "snapshotting". (FWIW, a month ago, when rfredsmith replied to one of my posts asking "what do you intend to do with the resulting snapshot?" I was at a loss to succinctly explain this problem/goal.) Performing cleanup ONLY immediately prior to (or as a step during) snapshotting isn't helpful toward minimizing on-demand root persistence file bloat... so I've built up a "hitlist" of likely/recurring paths where locale-specific helpdocs for various apps will accumulate, and I periodically run a bash script to perform batch deletion. I'll paste the paths hitlist into a followup pm.

Maybe it was/is obvious to everyone else that
remastering ~= snapshotting
or, said differently, that
"the snapshotting script is sufficiently flexible enough to accommodate
both generic remastering and creation of bootable, personal backups"
...but finding only "excluded from working_dir" (or removed from, in dzz's script) handling REALLY confused me.
"Do you want to preserve the content of your HOME in the snapshot?"
THAT is certainly understandable, as an exclusion consideration
but several of the outcommented "you might also consider excluding..." suggested items
left me wondering "Why aren't we cleaning suchandsuch in the SOURCE filesystem?"

a related observation:
Regardless whether pre-snapshot cleanup operations are performed from atop the snapshot script
or via a separate "cleanup" script, the snapshot script should (and neither refracta nor antix scripts currently do)
call updatedb immediately prior to creation of the squashfs.

To pursue additional weeding targets in the dzz "sid" iso (diminishing returns, but still significant IMO)
during live session I did the following:

( chmod +x ) ...or
$ apt-get remove mlocate
$ apt-get install mlocate
edit /(etc)/updatedb.conf and add /lib/live to the prunepaths line
PRUNEPATHS="/tmp /var/spool /media /lib/live"

$ updatedb

$ mlocate zh
this infrequent letter pair matches lang code for chinese
(I didn't, but) you might perform a separate "locate" pass looking for a few other locale -ish patterns:
zu_
uz_
xh_
(b/c a given pkg which writes locale-specific helpfiles might not include a "zh" set)

Browse the paths where "zh" files reside,
manually removing (via thunar, for instance) locale crud missed by bleachbit.
( for the moment, ignore /var/cache/man )
Take notes for each affected path (xxMb bloat) and, if possible, attempt to draft a "rule", er, pattern for use in the "cleanup" script.
F'rinstance:

# every time perl pkg is updated (like, every 2 days, eh) these get reinjected
rm /usr/share/winff/languages/winff.[^e][^n].po
rm /usr/share/tcltk/tcl8.5/msgs/[^e][^n]_[^g][^b].msg
rm /usr/share/perl/5.18.1/Unicode/Collate/Locale/*

# manually remove all except time-admin-en_GB.omf and time-admin-C.omf
# === ABOVE SUITS dzz BUT WOULD NOT SUIT A en_US USER ===
# 1Mb cruft here
# rm /usr/share/omf/time-admin/time-admin-*.omf

# traverse the subdirs, preserve only dir "C" and dir "en_gb"
# /usr/share/help

# (proofreading note: OOPS i pasted same path above, and below, and
# now i can't recall which path actually contains "translit*)
# dzz: preserve only these: "translit*", "C", "i18n", and "en_GB"
# /usr/share/help

# dzz: traverse each of the subdirs here, preserving only "C" and "en_GB" items
# /usr/share/omf

# can delete all SUBDIRS under po here
# (manual deletion, vs creating a slew of regex patterns to preserve various files)
# /usr/share/doc/ruby-gettext/examples/po

# 98kb
rm -Rf /usr/share/doc/ruby-gettext/examples/locale/*

# 100kb
rm /usr/share/pacpl/[^e][^n]_[^U]*

# only 40kb but...
rm /var/lib/gconf/defaults/\%gconf-tree-*.xml

# woohoo (rolleyes) 146kb...
rm /usr/share/mc/hints/mc.hint.*

# 2.1Mb
rm /usr/share/xfce4/xkb/flags/[^e][^n].svg

here I checked progress:
launch bleachbit and (only ask it to) empty trash + tmpfiles
6.7Mb localization files were in trash


moving onward...
$ apt-get install ncdu
This ncurses directory usage util (or "ranger" ncurses file manager)
lets you "walk" the filesystem to find bloat

unmount any extraneous drives, and
cd /
find -iname *.png -size +200k
find -iname *.jpg -size +200k

# 12.4Mb
/usr/share/xfce4/backdrops/ABM97_2_1440x900.png
/usr/share/xfce4/backdrops/ABM97_2_1440x900.png
given their WxH dimensions, these could/should be smaller than about 300Kb
? convert 'em to *.jpg or otherwise optimize 'em via a graphics app

/home/user/.mozilla/firefox/8tg5gm1t.default/
ah.... you've already nixed the 10Mb bleachbit-vacuumed places.sqlite

# 1.5Mb (presence of sample file to check audio playback is a nice touch, but...)
/home/user/MemphisMinnie-MyWashWomansGone.ogg

Do you actually USE all the installed fonts?
(and, wading thru "too many choices" in prefs dropdowns is a drag)
Instead of uninstalling packages, maybe hand weed the fonts you don't/won't use
estimated ~5-8Mb savings in /usr/share/fonts/truetype

# given a 96dpi display...
#...when the heck are you EVER going to display a 2.6" tall icon?
# 7.1Mb
rm /usr/share/icons/gnome/256x256/*
# note: this is REGENERATED each time apt-get updated versions of this icon pkg are installed

# 2.3Mb
rm /usr/share/icons/Tango/icon-theme.cache

# 2.2Mb ---- unsure whether I personally would nix these, but
rm /usr/share/icons/nuoveXT2/128x128/*
rm /usr/share/icons/nuoveXT2/96x96
rm /usr/share/icons/hicolor/256*256/*

Beyond the above (32Mb or so), I didn't find any other remarkable sources of non-locale bloat within your iso, dzz.
Attention to app selection (in terms of costly pkg dependencies) could yield a significantly smaller iso, but that's a separate can o' worms.
Last edited by thwak on Sun Oct 06, 2013 11:50 pm, edited 1 time in total.

Re: Sid, Refracta and Refracta-tools

Sun Oct 06, 2013 11:38 pm

As another anti-bloat preventative measure, add the following entries to the /(etc)/apt/apt.conf configuration file:

APT::Install-Recommends "0" ; APT::Install-Suggests "0" ;

Re: Sid, Refracta and Refracta-tools

Mon Oct 07, 2013 2:11 am

thanks, thwak, for testing the iso

I booted your "sid" iso to a live session and found that you can quickly eliminate AT LEAST 275Mb ("size on disk") worth of bloat.
I'm curious to hear what the reduction in squashfs/iso size will be.

Well you could (at your own risk) run my script on it, remaster it, test it and let us know the result.

The primarly source of bloat is from extraneous locale-specific files

For a personal snapshot that is true.. however this iso is also intended as a help towards development of official Refracta, which would need to include all locales if Refracta is to be international.. that is not bloat, we spent a lot of time already improving internationalisation.

I already said I am not in favour of altering actual deb package files beyond conf files.. if there is a compelling reason to do that, there are "clean" methods like dpkg-divert

For a personal snapshot, you do what you want, that's the whole point of these tools (I got mine too!)

However you made a few other points that are worth investigating.. one thing at a time,, for now I'm happy to get a sid iso, well under 700, without losing any functionality.

EDIT Although not a fan of bleachbit I ran it in "scan" mode only, as root in a live session with all boxes ticked.

Code:
Disk space to be recovered: 274.9MB

With all boxes ticked except localisations only two .Xauthority files in /tmp and one clipboard were identified
Code:
Disk space to be recovered: 12.3KB

Re: Sid, Refracta and Refracta-tools

Fri Dec 13, 2013 1:39 am

New (unofficial) sid iso at the same url. 696MB. All the latest Sid (except what apt-listbugs shows) and Refracta tools at 10 December. Different default icon theme (got bored with Tango).

The package and version list is posted separately as a text file. Anyone else doing Sid images and struggling to get size down might find that useful.

Re: Sid, Refracta and Refracta-tools

Thu Jan 02, 2014 2:06 pm

Made a new sid iso yesterday, it was 715MB. New kernel 3.12-1-686-pae and related stuff seem heavier. Back to the old dilemma, size vs functionality.

Run out of ideas what else to shave off. Looking at what large packages might have to go, an example is gimp, ~60 MB installed but saves only ~15 in squash. I use gimp in my travelling usb to process photos.

All that's left is to manually delete clamav databases main.cvd and daily.cvd, ~70MB. They are already compressed so are still much the same in xz squashfs. Unfortunately clamscan runs but gives no warning if the databases do not exist.

The database is updated (or replaced) by freshclam. A (live cd) user must have a network connection and run freshclam manually (or databases can be downloaded elsewhere and copied from a removable) How useful is a virus scan anyway without current definitions?

So out they went.. the iso rebuilt at 643MB.

BTW why does Refracta use /home/clamav which is not the normal default?

Re: Sid, Refracta and Refracta-tools

Thu Jan 02, 2014 3:35 pm

I put the clamav database in /home so that it could be updated if you're running a live-usb with a persistent /home. I'm not too sad to see clamav go - I've only used it once in the last three years.

I don't know what else could be removed to save size. I thought about dropping some locales, but when I check the list of countries for iso downloads at sf, I think about adding some locales. My sid build is only about 450mb, because I left out all the major apps that anyone might want to use. So I haven't run into the size stuggle. But I am struggling with what to include or not for a basic system.

Post your latest build, and I'll take a look at it. Newest one I see at your site is 131210.

Re: Sid, Refracta and Refracta-tools

Thu Jan 02, 2014 4:13 pm

It will take till tomorrow to upload the new iso (slow) The only other changes are package upgrades.

In the meantime here's the package list (with versions): http://paste.debian.net/73821/

It's the nearest I can get to the official Wheezy list allowing for actual changes in sid and a few preference-orientated (mine) changes and additions. That's what fits in 643MB without clamav database and the tweaks discussed earlier, like *.pyc which have caused no known problems after some time.

Dropping locales is a good solution for a personal snapshot, as thwak pointed out but not for an international and universal (as reasonably possible) general release.

I didn't actually purge clamav, it's probably insignificant for space-saving.

Sometimes space gets freed on dist-upgrade. Still, at 643 there is room to play with, ~150 uncompressed.

BTW I packaged newer gtk-youtube-viewer from source for sid, it's in the iso. The old version doesn't work.

EDIT new iso is now posted (same url) with package list and md5
Post a reply