Menu
▾
▴
Re: [Astlinux-users] Large number of Firewall entries
|
From: Michael K. <mic...@ip...> - 2023-01-05 04:14:33
|
Hi All
Added this script in custom-rules.conf and it seems to work well:
-----
ipset create -exist udp_sip_hosts hash:net
ipset flush udp_sip_hosts
ip_addresses=$(grep "host" /mnt/kd/asterisk/sip_peers.conf | sed "s/host=//")
echo "$ip_addresses" | sed -n -r -e "s/^([0-9][0-9./]+)([[:space:]].*|)$/add -exist udp_sip_hosts \1/p" | ipset restore
iptables -A EXT_INPUT_CHAIN -m set --match-set udp_sip_hosts src -p udp --dport 5060 -j ACCEPT
-----
Regards
Michael Knill
On 3/1/2023, 2:03 pm, "Lonnie Abelbeck" <li...@lo... <mailto:li...@lo...>> wrote:
Michael,
Controlling the client is not ideal.
Not sure if it is worth the trouble, but rather than removing the IP address, you could mark it with a unique prefix, like:
#block#1.2.3.4
Then add another sed one-liner for a different set-name that gets added in a new unique chain which gets added as -I to the INPUT chain, like the adaptive ban plugin does. This order will make a -j DROP for udp 5060 act before the conntrack states.
Lonnie
> On Jan 2, 2023, at 6:16 PM, Michael Knill <mic...@ip... <mailto:mic...@ip...>> wrote:
>
> Thanks Lonnie. Sorry you had to roll your sleeves up. :-)¡
>
> Yes that makes perfect sense and it did what you said when tested. I really should have known this but it caught me out. I did stop SIP traffic going out but it was the remote peer's OPTIONS pings that was holding it up.
> We will test parsing sip_peers.conf looking at host= to pick up all the IP Addresses on the system and add them to the ipset.
>
> One thing I was thinking is that if we are sending OPTIONS pings to all these peers from the softswitch then theoretically we should not need to create any firewall rules as the session will already be set up in conntrack. I tested it by turning off OPTIONS pings at both ends, waiting for conntrack to time out and then turn on OPTIONS pings at the peer end. It did not work until I turned on OPTIONS pings at the softswitch end whereby I could make and receive calls again.
>
> Is this a bit risky do you think? Can you think of any breaking scenarios?
>
> Regards
> Michael Knill
>
>
>
> On 3/1/2023, 9:07 am, "Lonnie Abelbeck" <li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>>> wrote:
>
>
> Hi Michael,
>
>
> I rolled up my sleeves, and gave this a test in my lab:
>
>
> -- snip /etc/arno-iptables-firewall/custom-rules --
>
>
> ipset_ext_input_allow()
> {
> local proto="$1" port="$2" set="$3" file="$4"
>
>
> if [ ! -f "$file" ]; then
> echo "[CUSTOM RULE] ipset_ext_input_allow: File not found: $file"
> return
> fi
>
>
> echo "[CUSTOM RULE] IPSet Pass EXT->Local for Proto: $proto, Port: $port, Set: $set, IPsetFile: $file"
>
>
> ipset create -exist $set hash:net
> ipset flush $set
>
>
> sed -n -r -e "s/^([0-9][0-9./]+)([[:space:]].*|)$/add -exist ${set} \1/p" "$file" | ipset restore
>
>
> ip4tables -A EXT_INPUT_CHAIN -m set --match-set $set src -p $proto --dport $port -j ACCEPT
> }
> ipset_ext_input_allow udp 5060 udp_sip_hosts /tmp/sip-whitelist.netset
>
>
> --
>
>
> -- /tmp/sip-whitelist.netset --
> ##
> 1.2.3.4 #test
> #10.10.50.1
> 10.10.50.55
> #10.10.0.0/16
>
>
> --
>
>
> It worked as expected. Restarting the firewall "arno-iptables-firewall restart" applies the current IPv4 .netset file.
>
>
>
>
>> If I then remove the address and restart the firewall, the address is removed from the list (ipset list confirms this) but the address is still open in the firewall. I cannot remove it unless I reboot the system.
>
>
> What you are seeing is the iptables conntrack state table, eventually the UDP state will expire after 120 seconds (unless traffic resets the state)
>
>
> Source Port (#'s) Destination Port Protocol Packets Bytes TTL
> 10.10.50.1 5060 10.10.50.64 5060 UDP 24 13856 1:29
>
>
> After the TTL counts down to 0 then the conntrack state disappears. The iptables conntrack state table makes the firewall much more efficient. This behavior has always existed.
>
>
> So in your testing, if you wait 2 minutes after you remove an IP and apply the change, the IP will be blocked for UDP 5060 traffic.
>
>
> If you are getting a constant stream of UDP 5060 traffic from that IP then you would need to take additional measures to block further traffic. For example, if you allowed a remote SIP endpoint to register more often than 120 seconds, removing the IP from the "allowed" ipset would not "block" it until the conntrack state disappears.
>
>
> Make sense?
>
>
> Lonnie
>
>
>
>
>
>
>> On Jan 2, 2023, at 2:26 PM, Michael Knill <mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>>> wrote:
>>
>> Hi Lonnie
>>
>> Thanks for this. Unfortunately I still need to reboot the system for it to reread the netset rules if I remove an ipset entry.
>> Here is my custom-rules.conf:
>> --------
>> ipset create -exist udp_sip_hosts hash:net
>> ipset flush udp_sip_hosts
>> ipset add -exist udp_sip_hosts <my ip address added here>
>> iptables -A EXT_INPUT_CHAIN -m set --match-set udp_sip_hosts src -p udp --dport 5060 -j ACCEPT
>> --------
>>
>> If I add another IP Address to the list as below and restart the firewall it works fine and I see it when I do an ipset list:
>> ipset add -exist udp_sip_hosts <1st ip address>
>> ipset add -exist udp_sip_hosts <2nd ip address>
>>
>> If I then remove the address and restart the firewall, the address is removed from the list (ipset list confirms this) but the address is still open in the firewall. I cannot remove it unless I reboot the system.
>> Obviously not workable I'm afraid.
>>
>> Regards
>> Michael Knill
>>
>>
>>
>> On 3/1/2023, 3:22 am, "Lonnie Abelbeck" <li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>> <mailto:li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>>>> wrote:
>>
>>
>> Hi Michael,
>>
>>
>> Referring to the "apply_ipset_netset()" function (here [1])
>>
>>
>> Add "-exist" to the "create" and "add" (man-page [2]) commands.
>>
>>
>> Note that you can create the ipset from a text file within the /etc/arno-iptables-firewall/custom-rules script. Edit your text file and reload the firewall.
>>
>>
>> Using "ipset create -exist ..." will not fail if the ipset already exists. "ipset flush ..." will clear any pre-existing ipset.
>>
>>
>> Tip -> I would probably use "hash:net" instead of "hash:ip" so you could use CIDRs if you wanted.
>>
>>
>> custom-rules script snippet
>> --
>> ipset create -exist udp_sip_hosts hash:net
>> ipset flush udp_sip_hosts
>>
>>
>> ## either a one-liner from a text file "sip-whitelist.netset"
>> sed -n -r -e "s/^([0-9][0-9./]+)([[:space:]].*|)$/add -exist udp_sip_hosts \1/p" sip-whitelist.netset | ipset restore
>>
>>
>> ## Or, loop getting IPv4s from a text file "sip-whitelist.netset"
>> ipset add -exist udp_sip_hosts <IP Address>
>> ## done-loop
>>
>>
>> iptables -A EXT_INPUT_CHAIN -m set --match-set udp_sip_hosts src -p udp --dport 5060 -j ACCEPT
>> --
>>
>>
>> Lonnie
>>
>>
>> [1] https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275 <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275&gt;>>
>>
>>
>> [2] https://ipset.netfilter.org/ipset.man.html <https://ipset.netfilter.org/ipset.man.html> <https://ipset.netfilter.org/ipset.man.html> <https://ipset.netfilter.org/ipset.man.html>> <https://ipset.netfilter.org/ipset.man.html> <https://ipset.netfilter.org/ipset.man.html>> <https://ipset.netfilter.org/ipset.man.html>> <https://ipset.netfilter.org/ipset.man.html&gt;>>
>>
>>
>>
>>
>>>> On Jan 1, 2023, at 11:44 PM, Michael Knill <mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>>>> wrote:
>>>
>>> Hi All
>>> Merry Christmas and Happy New Year.
>>>
>>> Just rejuvenating this thread as I am building our new softswitch and playing with ipset as you offered below.
>>> We have done the following:
>>>
>>> Using CLI:
>>> ipset create udp_sip_hosts hash:ip
>>> ipset add udp_sip_hosts <IP Address>
>>>
>>> In custom-rules.conf
>>> iptables -A EXT_INPUT_CHAIN -m set --match-set udp_sip_hosts src -p udp --dport 5060 -j ACCEPT
>>>
>>> It all seems to work fine but I obviously am an iptables noob as I have no idea what to do when I make changes to the ipset as it does not change even after a firewall restart.
>>> I'm sure there is something I need to do which will get iptables to reread the ipset?
>>>
>>> Thanks guys.
>>>
>>> Regards
>>> Michael Knill
>>>
>>>
>>>
>>>> On 27/9/2021, 10:54 am, "Lonnie Abelbeck" <li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>> <mailto:li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>>> <mailto:li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>> <mailto:li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>>>>> wrote:
>>>
>>>
>>> Michael,
>>>
>>>
>>> The /mnt/kd/arno-iptables-firewall/custom-rules is a basic shell script, so parsing sip.conf using 'sed' or such should be reasonably straightforward.
>>>
>>>
>>> BTW, for extra credit, if you combined all the allowed SIP IPs into an ipset (ex. udp_sip_hosts), you can very efficiently match all of them with only one rule:
>>> --
>>> iptables -A EXT_INPUT_CHAIN -m set --match-set udp_sip_hosts src -p udp --dport 5060 -j ACCEPT
>>> --
>>> That would allow you to rebuild only the "udp_sip_hosts" ipset when the sip.conf got changed, without rebuilding the firewall. Though requires some 'ipset' command knowledge, though not complex at all.
>>>
>>>
>>> Example 'ipset' usage in AstLinux:
>>> https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275 <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275&gt;>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275&gt;>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275&gt;>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275&gt;>> <https://github.com/astlinux-project/astlinux/blob/d95ba9c3914b135da4440cb95f32af61a41d4650/package/arnofw/aif/bin/arno-iptables-firewall#L4275&amp;gt;&gt;>>
>>>
>>>
>>> If you only use IPv4 a lot of the example can be simplified.
>>>
>>>
>>> Lonnie
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Sep 26, 2021, at 7:17 PM, Michael Knill <mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>>>>> wrote:
>>>>
>>>> Thanks Lonnie.
>>>>
>>>> Actually now that I think about it, is there any reason why the custom rule could not parse sip.conf for host=<IP Address> and open up all Public IP's?
>>>> It would mean that you would need to restart the firewall every time you modified sip.conf but I'm sure we could build this into our portal very simply.
>>>>
>>>> Regards
>>>> Michael Knill
>>>>
>>>> On 27/9/21, 9:47 am, "Lonnie Abelbeck" <li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>> <mailto:li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>>> <mailto:li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>> <mailto:li...@lo... <mailto:li...@lo...> <mailto:li...@lo... <mailto:li...@lo...>>>>> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> With 300 rules and the same across all your boxes, I would use /mnt/kd/arno-iptables-firewall/custom-rules to define these.
>>>>
>>>> Very similar to the deny_ext_local() example I posted recently, but the reverse ... pass_ext_local() using -j ACCEPT
>>>>
>>>> Without testing, something like ...
>>>> --
>>>> pass_ext_local()
>>>> {
>>>> local proto="$1" host="$2" port="$3"
>>>>
>>>> echo "[CUSTOM RULE] Pass EXT->Local for Proto: $proto, Host: $host, Port: $port"
>>>> iptables -A EXT_INPUT_CHAIN -s $host -p $proto --dport $port -j ACCEPT
>>>> }
>>>> ## uncomment to enable ##
>>>> #pass_ext_local udp 1.2.3.4 5060
>>>> #pass_ext_local tcp 1.2.3.0/24 5061
>>>> --
>>>>
>>>> If you only use udp/5060, you could simplify things, maybe only one "echo" statement and a variable defining all 300 IPs. Generic shell scripting.
>>>>
>>>> Again untested ...
>>>> --
>>>> pass_ext_local_udp_sip()
>>>> {
>>>> local host proto="udp" port="5060" IFS
>>>> local sip_hosts="1.2.3.4 1.22.33.40 1.22.33.41 1.22.33.42 1.22.33.43 1.22.33.44 1.22.33.45 1.22.33.46 1.22.33.47 1.22.33.48"
>>>>
>>>> echo "[CUSTOM RULE] Pass EXT->Local for UDP/5060 SIP Hosts"
>>>> unset IFS
>>>> for host in $sip_hosts; do
>>>> iptables -A EXT_INPUT_CHAIN -s $host -p $proto --dport $port -j ACCEPT
>>>> done
>>>> }
>>>> pass_ext_local_udp_sip
>>>> --
>>>>
>>>> Alternatively, you could define the sip_hosts variable with a file if desired.
>>>>
>>>> Lonnie
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On Sep 26, 2021, at 5:32 PM, Michael Knill <mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>>>>> wrote:
>>>>>
>>>>> Hi Group
>>>>>
>>>>> I'm looking to have a large number of firewall entries in Astlinux e.g. 300. They would be all the same e.g. I want to open port 5060 from multiple sites.
>>>>> Is there an easier/neater way to do this other than lots of firewall entries in the Firewall Tab?
>>>>>
>>>>> Regards
>>>>>
>>>>> Michael Knill
>>>>> Managing Director
>>>>>
>>>>> D: +61 2 6189 1360
>>>>> P: +61 2 6140 4656
>>>>> E: mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>> <mailto:mic...@ip... <mailto:mic...@ip...> <mailto:mic...@ip... <mailto:mic...@ip...>>>>
>>>>> W: ipcsolutions.com.au
>>>>>
>>>>> <image001.png>
>>>>> Smarter Business Communications
>>>>>
>>>>> _______________________________________________
>>>>> Astlinux-users mailing list
>>>>> Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>>>
>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&amp;gt;&gt;>>
>>>>>
>>>>> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>>>.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Astlinux-users mailing list
>>>> Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>>>
>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&amp;gt;&gt;>>
>>>>
>>>> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>>>.
>>>>
>>>>
>>>> _______________________________________________
>>>> Astlinux-users mailing list
>>>> Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>>>
>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&amp;gt;&gt;>>
>>>>
>>>> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>>>.
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Astlinux-users mailing list
>>> Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>>>
>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&amp;gt;&gt;>>
>>>
>>>
>>> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>>>.
>>>
>>>
>>> _______________________________________________
>>> Astlinux-users mailing list
>>> Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>>
>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>>
>>>
>>> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>>.
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Astlinux-users mailing list
>> Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>> <mailto:Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>>
>> https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>> <https://lists.sourceforge.net/lists/listinfo/astlinux-users&gt;>>
>>
>>
>> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>> <mailto:pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>>.
>>
>>
>> _______________________________________________
>> Astlinux-users mailing list
>> Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>
>> https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>>
>>
>> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>.
>
>
>
>
>
>
> _______________________________________________
> Astlinux-users mailing list
> Ast...@li... <mailto:Ast...@li...> <mailto:Ast...@li... <mailto:Ast...@li...>>
> https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users> <https://lists.sourceforge.net/lists/listinfo/astlinux-users>>
>
>
> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...> <mailto:pa...@kr... <mailto:pa...@kr...>>.
>
>
> _______________________________________________
> Astlinux-users mailing list
> Ast...@li... <mailto:Ast...@li...>
> https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users>
>
> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...>.
_______________________________________________
Astlinux-users mailing list
Ast...@li... <mailto:Ast...@li...>
https://lists.sourceforge.net/lists/listinfo/astlinux-users <https://lists.sourceforge.net/lists/listinfo/astlinux-users>
Donations to support AstLinux are graciously accepted via PayPal to pa...@kr... <mailto:pa...@kr...>.
|
