From: Scott C. <sc...@sc...> - 2023-05-12 05:19:49
|
<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"></div><div dir="ltr">I was kind of remembering that you wrote about this. I wonder if we could/should compile our own Postgres server to bump up that limit. At first blush, that sounds like a terrible idea but I could be convinced. </div><div dir="ltr"><br><blockquote type="cite">On May 11, 2023, at 4:14 PM, Joe Carlson <jwc...@lb...> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><meta http-equiv="Content-Type" content="text/html; charset=utf-8">Hello,<div class=""><br class=""></div><div class="">I cannot be done. Even though the documentation for a TEXT type in postgresql says it is ‘variable unlimited length', there is an internal limitation of the size of an allocated buffer. It’s in the documentation for the character data type <a href="https://www.postgresql.org/docs/current/datatype-character.html:" class="">https://www.postgresql.org/docs/current/datatype-character.html:</a></div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><span style="font-family: "Open Sans", sans-serif; font-size: 14.4px; font-variant-ligatures: normal; orphans: 2; widows: 2; text-decoration-thickness: initial;" class="">In any case, the longest possible character string that can be stored is about 1 GB.</span></blockquote></div><div class=""><br class=""></div><div class="">I recently ran into this problem myself and tried various ways around the limit by storing chunks and trying to concatenate with plpgsql functions, but that did not work. (<a href="https://www.postgresql.org/message-id/800...@lb..." class="">https://www.postgresql.org/message-id/80025ECD-44A6-454F-A4F9-784474B84952%40lbl.gov</a>).</div><div class=""><br class=""></div><div class="">You need to store residues as chunked pieces in a separate table and rely on your middleware code to split pieces on injest and concatenate on output.</div><div class=""><br class=""></div><div class="">Joe Carlson<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On May 9, 2023, at 10:22 AM, Cheng, Chun-Huai via Gmod-schema <<a href="mailto:gmo...@li..." class="">gmo...@li...</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Hi,<div class=""><br class="ContentPasted0"></div><div class="ContentPasted0">We have a big genome with large chromosomes that each of them is greater than 1G bp. We're having trouble loading them into the 'feature' table. We've tried the Tripal FASTA loader and a custom script, but both failed with some error (in Postgres v12 log: ERROR: invalid memory alloc request size 1161290884). Is there any way we can load the sequences for the genome into Chado?</div><div class=""><br class="ContentPasted0"></div><div class="ContentPasted0">Here's some length information about the genome:</div><div class="ContentPasted0"><br class=""></div><div class="ContentPasted0">chr1 1853204363</div><div class="ContentPasted0">chr2 1709916750</div><div class="ContentPasted0">chr3 1527935595</div><div class="ContentPasted0">chr4 1588398909</div><div class="ContentPasted0">chr5 1297479159</div><div class="ContentPasted0">chr6 1379031673</div><div class=""><br class="ContentPasted0"></div><div class=""><br class="ContentPasted0"></div><div class="ContentPasted0">Thank you very much for your help,</div><br class=""></div><div class="elementToProof" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">Chun-Huai Cheng</span><br class=""></div></div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Gmod-schema mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="mailto:Gmo...@li..." style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Gmo...@li...</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="https://lists.sourceforge.net/lists/listinfo/gmod-schema" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://lists.sourceforge.net/lists/listinfo/gmod-schema</a></div></blockquote></div><br class=""></div><span>_______________________________________________</span><br><span>Gmod-schema mailing list</span><br><span>Gmo...@li...</span><br><span>https://lists.sourceforge.net/lists/listinfo/gmod-schema</span><br></div></blockquote></body></html> |